aten-ipmi-tools/REVERSING_CLIENT.md

109 KiB

ATEN iKVM Reverse Engineering Notes

Reverse engineering notes for the ATEN IPMI KVM viewer, extracted from JAR iKVM__V1.69.21.0x0 and native library package liblinux_x86_64__V1.0.5.

Detailed protocol specifications are in separate files:


Table of Contents

  1. Package Contents
  2. Binary Overview
  3. Build Environment
  4. libiKVM64.so -- KVM Console Library
  5. libSharedLibrary64.so -- Virtual Media Library
  6. Java Application Layer
  7. Class Hierarchy and Vtables
  8. Security Analysis
  9. Key Data Structures and Globals
  10. Round 5 Verification Results
  11. Round 6 Final Sweep Results
  12. Areas for Further Investigation

Package Contents

original_binaries/client/
├── iKVM__V1.69.21.0x0/           Java classes + resources
│   ├── tw/com/aten/ikvm/          KVM viewer application
│   │   ├── KVMMain.class          Entry point
│   │   ├── KVMParam.class         Parameter parsing
│   │   ├── BoardInfo.class        Board identification
│   │   ├── ui/                    GUI components
│   │   │   ├── Viewer.class       Main window
│   │   │   ├── RemoteVideo.class  Video display + input (44 native methods)
│   │   │   ├── onScreenKeyboard.class
│   │   │   ├── MyKeyboardPanel.class
│   │   │   ├── OptionFrame.class  Settings dialogs
│   │   │   ├── AST2050OptionFrame.class
│   │   │   ├── HermonOptionFrame.class
│   │   │   ├── SettingPanel.class
│   │   │   ├── HotKeyFrame.class
│   │   │   └── UserListFrame.class
│   │   ├── jni/                   JNI bridge classes
│   │   │   ├── RMConnection.class Connection management (3 native methods)
│   │   │   └── RMDesktop.class    Desktop composition wrapper
│   │   ├── util/                  Utilities
│   │   │   ├── KeyMap.class       VK→HID keycode translation
│   │   │   ├── Converter.class    Data type conversion
│   │   │   ├── VideoRecorder.class Screen recording
│   │   │   └── ZipFile.class      ZIP handling
│   │   └── bean/                  Data beans
│   │       ├── KVMInfoRepository.class
│   │       ├── MouseMode.class, KeyboardMode.class
│   │       ├── ResolutionInfo.class, YUVMode.class
│   │       └── HotKey.class, KeyDef.class, etc.
│   ├── tw/com/aten/vstorage/      Virtual media storage
│   │   ├── VirtualStorage.class   Main VM class (16 native methods)
│   │   └── bean/VSBtnEventInfo.class
│   ├── tw/com/aten/bean/          Shared data beans
│   │   ├── ConnInfo.class         Connection info fields
│   │   ├── UserInfo.class         Username/password
│   │   ├── PlatformInfo.class     Platform identification
│   │   ├── VirtualUsbInfo.class   VM port/config
│   │   └── InfoRepository.class   Shared state
│   ├── ch/randelshofer/media/     Third-party AVI library
│   │   └── avi/AVIOutputStream.class  (for screen recording)
│   └── res/                       UI resources (keyboard layout PNGs)
│
└── liblinux_x86_64__V1.0.5/      Native shared libraries
    ├── libiKVM64.so               KVM protocol engine (204,592 bytes)
    └── libSharedLibrary64.so      Virtual media engine (261,688 bytes)

Binary Overview

libiKVM64.so

Property Value
Format ELF 64-bit LSB shared object, x86-64
Size 204,592 bytes (200 KB)
.text size 0x134D8 (78,040 bytes)
.rodata size 0x595B (22,875 bytes)
.data size 0x3450 (13,392 bytes)
.bss size 0x46F2610 (~74 MB)
Functions ~576 (Ghidra), 443 exported text symbols
JNI exports 47 (3 RMConnection + 44 RemoteVideo)
MD5 a5cd4f2be1f674a0251c14576412c568
SHA256 46897ab5bdaacf45f42acdaa0e95afdbeb83c06147c0cb6d78c9d4cb8eeb6df5

libSharedLibrary64.so

Property Value
Format ELF 64-bit LSB shared object, x86-64
Size 261,688 bytes (256 KB)
.text size 0x21BC8 (138,184 bytes)
.rodata size 0xB7B (2,939 bytes)
.data size 0x17E0 (6,112 bytes)
.bss size 0x21370 (136,048 bytes)
Functions ~722 (Ghidra), 341 exported text symbols
JNI exports 16 (VirtualStorage)
MD5 8a02baead64cf2947c372b1218fa1751
SHA256 32bc5f23f479e4d6fa5033d31ee90a5e8cdc46104c511dfbc588c5c93950025a

Notable .bss size difference

libiKVM64.so has a ~74 MB .bss section, primarily due to:

  • Video decoder buffers (ast_Buffer at BSS, various YUV/Huffman tables)
  • Framebuffer memory (1920x1200x4 = ~9 MB + decoder working buffers)
  • Multiple Huffman decode tables (HTAC, HTDC at ~384 KB combined)
  • Color conversion lookup tables (Cr_tab, Cb_tab, Cr_Cb_green_tab)

Build Environment

Property Value
Compiler GCC 3.4.6 20060404 (Red Hat 3.4.6-10/11)
Target x86-64 Linux, System V ABI
C++ ABI CXXABI_1.3, itanium-style name mangling
Linking Dynamic, stripped (no debug symbols)
Symbols C++ mangled names preserved in .dynsym
File dates February 3, 2015

Both binaries were compiled with a very old GCC (3.4.6 from RHEL/CentOS 4 era). C++ exceptions are used (__cxa_throw, __cxa_begin_catch, personality routines). The code uses the older C++ ABI conventions.

Shared Library Dependencies

Both libraries link to:

  • libstdc++.so.6 -- C++ standard library
  • libm.so.6 -- Math library
  • libgcc_s.so.1 -- GCC runtime
  • libc.so.6 -- C standard library

libiKVM64.so additionally uses X11 functions loaded at runtime:

  • XOpenDisplay, XCloseDisplay
  • XkbGetIndicatorState (keyboard LED state for CapsLock/NumLock/ScrollLock)
  • XKeysymToKeycode, XTestFakeKeyEvent

libiKVM64.so -- KVM Console Library

JNI Interface

RMConnection (3 methods at tw.com.aten.ikvm.jni.RMConnection):

Export Address Purpose
Java_..._RMConnection_init 0x00119950 Initialize JNI field IDs for ConnInfo/UserInfo
Java_..._RMConnection_keepActive 0x00119a80 Establish TCP connection via TcpSocket
Java_..._RMConnection_checkValidUser 0x00119c60 Authenticate (triggers RFB handshake)

RemoteVideo (44 methods at tw.com.aten.ikvm.ui.RemoteVideo):

Export Address Category
init 0x0011fbd0 Initialize video system, JNI callbacks, X11 LED state
destory [sic] 0x0011ffd0 Cleanup/destruction
runImage 0x001201e0 Receive + decode one frame
updateImage 0x00120050 Copy framebuffer to Java ByteBuffer
getDecodeImage 0x001207b0 Get decoded image data
keyboardAction 0x00121000 Send keyboard event
mouseAction 0x001212d0 Send mouse event
catchLoop 0x001216c0 No-op (empty function body)
doCatch 0x00121570 Set filterFlag global (no message processing)
refresh 0x00121510 JNI buffer management (MonitorEnter/Exit, no network I/O)
syncMouse 0x00121500 Send MouseSync
sendKeepAliveAck 0x00121a00 Respond to server keepalive
sendPrivilegeCtrl 0x00121970 Send privilege control
hotPlug 0x00121a30 Mouse hot-plug reset
setPowerOn/Off/Reset 0x00121a50-b0 Power control
setScreenUILang 0x00121b00 Set OSD language
setQosParameter 0x00121c00 Set video QoS
changeLEDstate 0x00120cd0 Local X11 key sim (no network I/O)
getLEDstate/setLEDstate 0x00120de0/e60 LED state get/set
screenRecording* 0x001216f0+ Screen capture to AVI
get/setMouseMode 0x00121470/a0 Mouse mode control
getFPS/resetFPS 0x00121400/f0 FPS counter
getScreenLevel/Mode 0x00121410/40 Screen info query
getPlatVer 0x001216e0 Get platform version
changeScreenInfo 0x00120c00 ScreenGetRect+ScreenSetInfo (type 0x32 if changed)
updateInfo 0x00120be0 Calls MouseUpdateInfo (sends type 0x37)
setReplayRecord 0x00120bc0 Set replay recording flag
getPowerStatus 0x00121ad0 Misnamed: calls MouseGetPT() (returns mouse config, not power)
getScreenUILang 0x00121b50 Query OSD language
stopCatch 0x001216d0 Stop message receive loop

JNI Callbacks (C → Java, registered in init):

Callback Signature Purpose
changeResolution (II)V Notify Java of resolution change
errorHandler (I)V Report errors
genKeyEvent (II)V Generate synthetic key events
privilegeCtrl (II[B)V Privilege control notification
setViewerConfig (I[B)V Pass viewer config to Java
addClipBounds (IIII)V Report dirty rectangles for repaint
getScreenUILangConf (II)V OSD language config
getQuickCursorUpdate (I)V Quick cursor update notification
setNormalStart (I)V Normal session start notification

C++ Class Architecture

                        TcpSocket
                            │
                        StreamSocket
                            │
                        NtwStream ─── Buffered network I/O
                            │
                        RMProtocol (abstract base)
                            │
                        RFBProtocol ─── RFB 3.8 protocol handler
                            │
    ┌───────────────────────┼───────────────────────┐
    │                       │                       │
RMDesktop              RMConnection             RMAuth
(composition)          (connect/auth)
    │
    ├── RMKeyboard → RFBKeyboard
    ├── RMMouse    → RFBMouse
    ├── RMScreen   → RFBScreen
    ├── RMPrivilege→ RFBPrivilege
    └── RMCryto    → RFBKMCryto

                        RMDecoder (abstract base)
                            │
    ┌───────────────┬───────┴───────┬──────────────┐
    │               │               │              │
ASTVideoDecoder  HermonVideo     YarkonVideo    Pilot3Video
    │            Decoder          Decoder        Decoder
    ├── ast2100 (AST2050)
    │   └── ASTDecode
    └── ast_jpeg (AST2100+)

Additional:  RMStorage, RMMisc, ErrMsg (exception type, 0x44 bytes: int + char[64])

Object sizes (from operator_new calls in constructors):

Class Allocation Size Notes
RMConnection 0x10 Contains vtable + RFBProtocol*
RFBProtocol 0x70 Protocol state, fw_protocol_flag at 0x48
RMDesktop 0x30 Composition: keyboard+mouse+screen+privilege ptrs
RFBKeyboard 0xB8 Includes 3 std::map lookup tables for VK->keysym
RFBMouse 0x28 Includes RFBKMCryto* for mouse encryption
RFBScreen 0x2080 Large: includes cursor data, frame refs
RFBPrivilege 0x120 Includes 256-byte key material buffer + RFBKMCryto*
RFBKMCryto 0x2110 AES state: key schedule, S-boxes, round keys
ASTVideoDecoder 0x58 Wrapper for ast2100 or ast_jpeg
HermonVideoDecoder 0x58 Hermon chipset decoder
YarkonVideoDecoder 0x60 Yarkon chipset decoder
Pilot3VideoDecoder 0x458 Pilot III chipset decoder
ast_jpeg 0x588 JPEG-variant decoder (AST2100+)
ast2100 0x1029E8 VQ + Huffman decoder (AST2050), ~1 MB
ErrMsg 0x44 Exception: int error_code + char[64] message

Video Decoder Architecture

Each decoder class inherits from RMDecoder and implements:

  • Decode(UpdateBlocks_t&) -- Main decode entry point (virtual)
  • GetDecodedFrame() -- Return pointer to decoded pixels (virtual)
  • MixedCursor(CursorInfo) -- Composite hardware cursor (ASTVideoDecoder)

decoder_fun is a shared utility class used by the JPEG-like decoders:

  • IDCT_transform -- Inverse DCT (two implementations at 0x0010dff8, 0x001144f0)
  • process_Huffman_data_unit -- Huffman decode
  • init_JPG_decoding, init_jpg_table -- JPEG table init
  • get_DCT, get_codebuf, get_newbits, get_newbuf -- Bitstream helpers
  • get_CrtoR, get_limittable -- Color space helpers
  • calculate_tabs -- Quantization table calculation
  • WORD_hi_lo -- Byte swap utility

AST2050 (encoding 0x57) via ast2100 class:

  • Contains VQ_Decompress -- Vector Quantization decompression
  • Contains RC4_crypt (at 0x0011b8f0) -- apparently uses RC4 in VQ decode
  • InitParameter, init_QT, getKbits -- Initialization and bitstream

AST2100+ (encoding 0x58) via ast_jpeg class:

  • Pure JPEG variant decoder
  • YUVToRGB (at 0x0010e748) -- Color space conversion
  • PsudoStreamSwap16/32 -- Byte order helpers
  • get_decoderID, get_buffer, get_ImageData -- Accessor methods
  • setScreenInfo -- Resolution configuration
  • updatereadbuf -- Buffer management

Hermon (encoding 0x59) via HermonVideoDecoder:

  • ConvertVierwerPixelFormat [sic] -- Pixel format conversion
  • Constructor takes (Decodeinfo, byte*, byte*) parameters

Yarkon (encoding 0x60) via YarkonVideoDecoder:

  • ConvertVierwerPixelFormat -- Pixel format conversion (same typo)

Pilot3 (encoding 0x61) via Pilot3VideoDecoder:

  • SetRect -- Rectangle operations

NtwStream -- Network I/O Layer

Buffered network stream with batched writes. Reads are unbuffered (each StreamRead call goes directly to recv() via TcpSocket). The 1520-byte internal buffer is exclusively for write batching.

NtwStream object layout (allocated size = 0x628 bytes):

Offset   Size   Type              Field              Notes
0x00     0x28   pthread_mutex_t   write_mutex        Protects write buffer
0x28     0x5F0  uchar[]           write_buffer       1520-byte write batch buffer
0x618    8      uchar*            write_ptr          Current position in write buffer
0x620    8      TcpSocket*        socket             Underlying TCP socket (0x60 bytes)

Constructor at 0x0010efb0: allocates TcpSocket (0x60 bytes), stores at offset 0x620. Initializes the pthread mutex at offset 0x00. Sets write_ptr (offset 0x618) to point to the write buffer start (this + 0x28).

Method Address Purpose
StreamRead(buf, len) 0x0010f0a0 Read exact bytes from socket (loop until all received)
StreamRead8() 0x0010f0f0 Read 1 byte
StreamRead16() 0x0010f110 Read 2 bytes (big-endian)
StreamRead32() 0x0010f140 Read 4 bytes (big-endian)
StreamReadSkip(n) 0x0010f180 Skip n bytes (allocates temp buffer, wasteful)
StreamWrite(buf, len) 0x0010f1f0 Buffer write data (flushes + sends direct if overflow)
StreamWrite8(val) 0x0010f340 Buffer 1 byte
StreamWrite16(val) 0x0010f300 Buffer 2 bytes (big-endian)
StreamWrite32(val) 0x0010f2b0 Buffer 4 bytes (big-endian)
StreamWriteStart() 0x0010f370 Begin write batch (pthread_mutex_lock only)
StreamWriteFlush() 0x0010f380 Flush buffer via socket write + pthread_mutex_unlock
StreamWriteSkip(n) 0x0010f3f0 Write n zero bytes (allocates temp buffer, wasteful)
StreamRecv() 0x0010f4b0 Single raw recv() call (no retry loop)
Connect() 0x0010f080 Delegates to TcpSocket::Connect() via vtable
PsudoStreamSwap16() -- Byte-swap u16 (host to network order)
PsudoStreamSwap32() -- Byte-swap u32

Write batching protocol: StreamWriteStart() locks the mutex → multiple StreamWrite*() calls accumulate into buffer via memcpyStreamWriteFlush() sends all buffered data in one socket->write() call and unlocks the mutex. This ensures protocol messages are sent atomically.

TcpSocket -- TCP Socket Layer

TcpSocket object layout (allocated size = 0x60 bytes):

Offset   Size   Type         Field              Notes
0x00     0x08   void**       vtable             VFT pointer
0x08     0x34   char[52]     hostname           Copied via strcpy (overflow risk)
0x3C     0x04   int          port               TCP port number
0x40     0x04   int          record_flag        Debug: record recv'd data to file
0x44     0x04   int          playback_flag      Debug: read from file instead of socket
0x48     0x08   FILE*        record_file        File handle for recording
0x50     0x08   FILE*        playback_file      File handle for playback
0x58     0x04   int          socket_fd          Connected socket FD
0x5C     0x04   int          is_server          1=listen/accept mode, 0=connect mode

TcpSocket vtable (at 0x0022f750):

Offset Address Method
+0x00 0x00117b70 ~TcpSocket() D1 destructor
+0x08 0x00117b20 ~TcpSocket() D0 destructor
+0x10 0x00117a80 Connect()
+0x18 0x00117310 read(buf, len) -- uses select(30s) + recv()
+0x20 0x00117530 write(buf, len) -- uses select(30s) + send()

Socket options set in CreateScok (0x001179c0, note typo):

  • SO_SNDBUF = 0x40000 (256 KB)
  • SO_RCVBUF = 0x40000 (256 KB)
  • enableNagles() is a no-op (returns 1) -- TCP_NODELAY is NOT set
  • No SO_KEEPALIVE set
  • Connection via getaddrinfo() + socket() + connect(), SOCK_STREAM

Timeout: Both read() and write() use select() with 30-second timeout (tv_sec = 0x1e). On timeout or error, throws C++ ErrMsg exception. Error codes: -1 = socket closed/failed, -2 = timeout.

SIGPIPE handler: BrokenPipe_handle prints message and returns (prevents JVM crash). Registered via InitSignal() at 0x001170e0.

Connection teardown: No graceful disconnect message. TcpSocket::EndSock (0x00117aa0) calls shutdown(fd, SHUT_RDWR) then close(fd).

File playback mode: TcpSocket has built-in debug support for reading from/writing to files instead of sockets, controlled by record_flag and playback_flag. Used for session replay/recording.

Protocol Handler Dispatch

RFBProtocol::ProtocolHandler() at 0x001183d0 dispatches server messages by type byte using a relative jump table at 0x00127790 (61 entries, 0x00-0x3C).

Each jump table entry is a 32-bit signed offset relative to the table base address 0x00127790. The default handler (54 entries) is at 0x001183f5 and simply returns the type byte to the caller. All non-default handler code is inline within the ProtocolHandler function body (not separate functions).

ProtocolHandler operating modes (controlled by this->0x30):

  • Mode 0: Normal operation. Clears screen update state, reads type byte, dispatches via jump table.
  • Mode 1: Replay/capture mode. Copies this->0x38 (source file pointer) to the screen decoder context, sets capture flag, reads type byte, dispatches.
  • Other: Reads type byte directly (no state setup), dispatches.

Complete jump table (7 non-default handlers, all inline at 0x001184xx):

Type Hex Jump Target Handler Called Wire Bytes After Type
0 0x00 0x0011849a RFBScreen::ScreenDecode (0x0010E7E8) 3+2+2+2+2+4+4+4+data
4 0x04 0x001184de RFBScreen::ScreenCursorPosProc (0x0010E7F8) 4+4+4+4+4 [+4+pixels]
22 0x16 0x00118488 NtwStream::StreamRead8 (inline, reads 1 byte) 1 byte
53 0x35 0x0011844d RFBKeyboard::ProcKeyboardInfo (0x0011A720) then 2+3 = 5 bytes
RFBMouse::ProcMouseInfo (0x001198C0) (fallthrough)
55 0x37 0x00118457 RFBMouse::ProcMouseInfo (0x001198C0) 3 bytes
57 0x39 0x00118479 RFBPrivilege::ProcPrivilegeInfo (0x0011A120) 4+4+256 = 264 bytes
60 0x3C 0x0011846a RFBScreen::GetScreenUILang (0x00118CB0) 4+4 = 8 bytes

Type 0x35 and 0x37 relationship: Type 0x35 calls ProcKeyboardInfo (reads 2 bytes: mode, type) then falls through to the same code that type 0x37 jumps to (ProcMouseInfo, reads 3 bytes: mode, type, extra). So type 0x35 is a combined keyboard+mouse info message (5 bytes after type), while type 0x37 is mouse-only (3 bytes after type).

Type 0x00 capture mode: When the this->0x20 & 1 flag is set (capture mode), the type 0x00 handler also writes the received data to a capture file using fwrite().

Encryption (RFBKMCryto)

AES-128-CBC for mouse event encryption (keyboard is always cleartext):

Method Address Purpose
RFBKMCryto() 0x001148f0/0x00114920 Constructor (C2/C1 variants)
~RFBKMCryto() 0x00114950/0x00114970/0x00114990 Destructors (D2/D1/D0)
aes_gen_tables() 0x001149d0 Generate AES S-boxes and tables
aes_set_key() 0x001149e0 Set AES key from key material
aes_encrypt() 0x0010eaf8 (DEFAULT) AES block encrypt
aes_decrypt() 0x00115d50 AES block decrypt
SW_AES_ECB() 0x00116be0 Software AES-ECB: mode=0 encrypt, mode=1 decrypt
SW_AES_CBC() 0x0010e2e8 (DEFAULT) Software AES-CBC (used for mouse events)
RFB_AES128_EventCryto() 0x00116e30 High-level event encrypt: hardcoded key+IV
SetCipherKey() 0x001149b0 No-op stub: returns 1 without doing anything
EnCryto() 0x00117050 Encrypt interface: calls RFB_AES128_EventCryto
DeCryto() 0x001149c0 Decrypt interface
RSb 0x001225e0 AES inverse S-box table

The implementation is a complete software AES with key expansion, S-box generation, and CBC mode -- no dependency on OpenSSL or system crypto.

Key insight: SetCipherKey() is a no-op (returns 1 immediately). The AES key used for mouse event encryption is entirely hardcoded in RFB_AES128_EventCryto() -- it is the NIST AES-128 test vector:

Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C
IV:  00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

The function also contains hardcoded expanded round key constants on the stack. The server-provided 256-byte key material from PrivilegeInfo is NOT used for native-layer encryption -- it is forwarded to Java only.

SW_AES_ECB (at 0x00116be0): Parameters: (this, mode, keysize, input, num_blocks, key, output). Mode 0 = encrypt, mode 1 = decrypt. Key size is keysize * 64 + 128 bits. Processes blocks sequentially (no chaining).

SW_AES_CBC: Wraps SW_AES_ECB with CBC chaining (XOR with IV/previous ciphertext for encryption; decrypt then XOR for decryption).


libSharedLibrary64.so -- Virtual Media Library

JNI Interface

VirtualStorage (16 methods at tw.com.aten.vstorage.VirtualStorage):

Export Address Purpose
JASWInit 0x00114c18 Initialize VM subsystem (platformVer, companyId, boardId)
JASetUNamePwdIPPort 0x00114e74 Set credentials and target BMC
VUSPlugIn 0x00114aac Mount virtual device
VUSPlugOut 0x00114b78 Unmount virtual device
StartDev1Thread 0x00114876 Start device 1 protocol thread
StartDev2Thread 0x00114892 Start device 2 protocol thread
StartDev3Thread 0x001148ae Start device 3 protocol thread
StartCheckDevExistThread 0x0011485e Device presence check thread
StartShowMsgThread 0x001148ca GUI message display thread
StartChangeGUIObjectThread 0x0011496c GUI state update thread
RefreshDevCBContents 0x00114a48 Enumerate available devices
ChangeCBSelect 0x00114bb6 Change device selection
SetEachVMLanPort 0x00114bda Set per-device LAN port
JThterminate 0x00114c00 Terminate all threads
ResetLANDisconnect 0x00114f82 Reset after network disconnect

Also exports the global struct:

  • java_vsui at 0x0023f560 (0x2A0 bytes) -- shared Java↔native state

Architecture

libSharedLibrary64.so is a standalone C library (no C++ class hierarchy like libiKVM64.so). It uses function pointer tables for platform abstraction and global structures for state management.

Key Function Groups

Protocol Engine:

Function Address Purpose
VMSM_USB_BulkOnly 0x0011b922 Main USB Mass Storage state machine
State_Fn_RX_PDU_TAG 0x0011b24c PDU header parser, type/CBW dispatch
State_Fn_EXECUTE_CBW 0x0011ab24 SCSI command execution state
State_Fn_RX_CBW 0x0011ac68 CBW parsing state
State_Fn_RX_BULK_OUT_DATA 0x00117cc8 Bulk OUT (write) data receive state
State_Fn_RX_MOUNT_DEV_STATUS 0x0011b694 Mount status response handler
SetState_ReadPDU 0x00117b92 Reset state to 0x01, expect 8-byte header
Execute_VM_CMD 0x00119c06 SCSI command dispatch
FillUSBPlugInPkt 0x001167c0 Build plug-in packet with USB descriptors
FillSetEPCMDPkt 0x00116234 Build endpoint configuration
FillCSW 0x00119fe8 Build Command Status Wrapper
FillDataPDUTag 0x00119d10 Build data PDU header
FillKeepAlivePkt 0x0011b1c2 Build keep-alive response
FillErrorData 0x00119f5a Build error response (CSW status from ASC/ASCQ)
FillDevID 0x001161d4 Build device identification
FillHostDescriptorData 0x0011cd4a Build host descriptor data
Core_VM_SendUnMountPkt 0x0011bdf2 Send unmount
RefreshBtnMode_Fill 0x0011c4d2 GUI button state

Mount Methods:

Function Address Purpose
Core_Mount_VM 0x0001f8ea Main mount orchestration
MtMethod_Media 0x0011df42 Direct TCP mount (primary)
MtMethod_WebISO 0x0011e946 HTTP ISO upload mount
MtMethod_UploadIMA 0x0011e14a HTTP floppy upload mount
UnMtMethod_WebISO 0x0011db14 HTTP ISO unmount
DetermineTCPType 0x0001f76a Choose IPv4 vs IPv6
CheckVMInfoBetSWAndFW 0x0001cc7e Validate VM config with firmware

Encryption:

Function Address Purpose
Encrypt_RC4 0x0011e0a6 RC4 encrypt wrapper
RC4_init 0x00115c10 RC4 Key Scheduling Algorithm
RC4_byte 0x00115cf2 RC4 single byte
RC4_encrypt 0x00115d88 RC4 Pseudo-Random Generation Algorithm

ISO/CD-ROM SCSI Handlers:

Function Address SCSI Command
IsoCommand 0x0012a9da Main dispatch (switch on opcode)
Inquiry 0x00128478 INQUIRY (0x12)
Startstopunit 0x00128aa0 START STOP UNIT (0x1B) -- eject/load emulation
MediumRemoval -- PREVENT ALLOW MEDIUM REMOVAL (0x1E)
ReadCapacity 0x0012a082 READ CAPACITY (0x25)
Read10 0x00129f42 READ(10) (0x28)
Read12 0x0012a868 READ(12) (0xA8) -- stub, returns 30 zero bytes
ReadBuffer 0x0012a7aa READ BUFFER (0x3C) -- always returns error
ReadToc 0x00129ba4 READ TOC (0x43)
ReadCD -- READ CD (0xBE)
ModeSense 0x00128f56 MODE SENSE(10) (0x5A)
GetConfig 0x00127378 GET CONFIGURATION (0x46)
GetEventStatus 0x00129962 GET EVENT STATUS NOTIFICATION (0x4A)
ReadFormatCapacities 0x0012a1d4 READ FORMAT CAPACITIES (0x23)
ReadDiscInformation 0x0002a378 READ DISC INFORMATION (0x51)
ReadBufferCapacity 0x0002a708 READ BUFFER CAPACITY (0x5C)
ReadMediaSerialNumber 0x0002a7f4 READ MEDIA SERIAL NUMBER (0xAB)
SetReadAhead 0x00128378 SET READ AHEAD (0xA7) -- validates LBA ordering
ReportKey 0x00029e16 REPORT KEY (0xA4)
SetStream 0x00028edc SET STREAMING (0xB6)
SetCDSpeed 0x00128e80 SET CD SPEED (0xBB) -- returns 32 zero bytes
MachanismStatus 0x001288be MECHANISM STATUS (0xBD) -- returns 8-byte status
Test 0x0002a8dc TEST (0x01)
TestUnitReady 0x00128f10 TEST UNIT READY (0x00)
RequestSense 0x001282d6 REQUEST SENSE (0x03)
Zero 0x00029b72 Zero-fill utility
ParOK 0x000271fe Parameter validation
EventDes 0x00029b42 Event descriptor builder

Floppy/IMA Handlers:

Function Address Purpose
FileStorCMD_IMA_API 0x00018380 IMA file SCSI command API
FileStorDevCMDExecuteIMA_Default 0x00017fea Default IMA command execution
FileStorDevCMDExecuteISO_Default 0x0001879a Default ISO command execution
FileStorDevCMDExecuteISO_FixInquiry 0x00018d6a ISO INQUIRY fixup
FileStorCMD_ISO_API 0x00019b52 ISO file SCSI command API
CalculateISOFileBlockNum 0x00115e20 Calculate ISO sector count

Physical Device Handlers (pass-through to real hardware):

Function Address Purpose
Linux_PhyStorDevCMDExecute_Default 0x00122902 Default physical device passthrough
Linux_PhyStorDevCMDExecute_IDECDROMFixInquiry 0x00123218 IDE CD-ROM INQUIRY fixup
Linux_PhyStorDevCMDExecute_USBCDROMFixInquiry 0x00122fd2 USB CD-ROM INQUIRY fixup
Linux_PhyStorDevCMDExecute_SimToFloppy 0x001151ac Simulate floppy from physical device
Linux_PhyStorDevCMDExecute_SimToFlash 0x00115746 Simulate flash from physical
Linux_PhyStorDevCMDExecute_SASHDFixInquiry 0x00123650 SAS/HDD INQUIRY fixup
Linux_PhyStorDevCMDExecute_HD_SimToRemovableDisk 0x00124a4e HDD as removable disk
Linux_PhyStorDevCMDExecute_HD_SimToRemovableDisk_logical 0x00124edc Logical volume
Linux_PhyStorDevCMDExecute_SASHD_SimToRemovableDisk_logical 0x00125124 SAS logical volume
Linux_PhyStorDevCMDExecute_IDECDROM_ReadFile 0x00123a80 IDE CD-ROM file read

Physical Device Scanning:

Function Address Purpose
Linux_ScanPhyStor_IDE_Floppy 0x00124570 Scan IDE floppy devices
Linux_PhyStorDevCheckDevType_IDEFloppy 0x001221e4 Check IDE floppy type
Linux_PhyStorDevCheckDevType_USBFloppy 0x00122114 Check USB floppy type
Linux_FileStorDevOpenISO 0x0012485c Open ISO file
Linux_FileStorDevCheckISOFileName 0x00124778 Validate ISO filename

Virtual Folder Mount (FAT filesystem emulation):

Function Address Purpose
TFATFileSystem_Open 0x0002b4b4 Open FAT filesystem
TFATFileSystem_Format 0x0002be82 Format virtual FAT volume
TFATFileSystem_DiskIO 0x0002b760 Disk I/O operations
VirImgFATFileSysIO 0x0002b85a Virtual image FAT I/O
TFATFileSystem_FAT* various FAT table manipulation
TFATFileSystem_Dir* various Directory operations
TFATFileSystemImage_* various Image file operations
Folder_RemoveImage 0x0002af6a Remove folder image
Linux_TFATFileSystemImage_GetAllFiles 0x00030a30 Enumerate host files
Linux_TFATFileSystem_DirGetFromWin32 0x0002fe24 Convert host dir to FAT
ConvertToDOSFormat 0x00031cc2 Convert filenames to 8.3

The folder mount feature creates a virtual FAT16 filesystem image from a host directory, emulating a removable USB disk.

Network Layer:

Function Address Purpose
Linux_VMTCPConnect_IPv4_Default 0x00121b02 IPv4 TCP connect
Linux_VMTCPConnect_IPv6_Default 0x00121d4c IPv6 TCP connect
Linux_VMTCPClose_IPv4_Default -- IPv4 close
Linux_VMTCPTx_IPv4_Default -- IPv4 send
Linux_VMTCPRx_IPv4_Default -- IPv4 receive
(IPv6 variants) -- IPv6 equivalents

Error Handling:

Function Address Purpose
ErrorHandle_Init 0x0001cb2e Initialize error handler
ErrorHandle_Set 0x0001cb5a Set error state
ErrorHandle_Get 0x0001cb88 Get error state
ErrorHandle_Handle 0x00015f4c Process error
ErrorHandle_DevSync 0x000160ea Device sync on error

Utility Functions:

Function Address Purpose
ByteToDWord 0x00015dec Byte array to 32-bit word
CMDSetTranslate 0x0001d090 Command set translation
DetectFileStorDevCMDType 0x0001d16c Detect file-backed device command set
DetectHostCMDType 0x0001d1fa Detect host device command set
DetectPhyStorDevCMDType 0x0001d0aa Detect physical device command set
GetDevCMDSet 0x0001d208 Get device command set
FunIDToDevInfo 0x0001448c Map function ID to device info
FunIDToHostDevType 0x0001450e Map function ID to host device type
FunIDToOEMStr 0x00014700 Map function ID to OEM string
GetRandomTimeStamp 0x00017860 Generate random timestamp (SID + HTTP boundaries)
InitVMSW 0x001178c8 Clear st_VMMainInfo, init queues, generate SID
VMInfoCalloc 0x00120bb2 Allocate per-device structures: calloc(devCount+1, 0x15920)
UI_PrepareVMResource 0x0011667c Init config tables, alloc per-device memory, set media types
StoreVMInfoFromGUI 0x0011d28c Validate + copy credentials (username<17, password<21)
UI_Mount_VM 0x0011f958 Full mount orchestration (check busy, store info, connect)
SetupVMInfoBetSWAndFW 0x0011ced4 Pre-mount setup: descriptors, HTTP port, persistent TCP
UI_PreVerifyVMInfo 0x0011ef0a Credential pre-verification (bit 6 verify-only path)
Core_GetDevStatusFromFW 0x0011741e Device status query via temp TCP (type=0x08)
GetHttpPortFromFW 0x0011cdd4 HTTP port query via temp TCP (type=0x0A, leaks socket)
UI_SetUSBPlugInPktEncrypt -- Set RC4 enable flag (DAT_0023f803)
UI_SetUSBPlugInPktSIDAuth -- Set SID auth flag (DAT_0023f804)
VM_Thread 0x0011baa4 Main per-device protocol loop (blocking I/O)
VM_Thread_VarInit 0x0011b9cc Initialize thread state vars
MtVM_Engine 0x0011de84 Mount dispatch engine (by device type)
UnMtVM_Engine_Normal 0x0011d48c Unmount dispatch engine
UnMountStatusInit 0x0011b5d0 BMC-initiated unmount handler
VM_TCPSocket_Terminate 0x00115f0c Close socket and reset flags
UI_GetDevStatusFromFW 0x001170fe Multi-slot device status query
GUIAtbQueueAPI 0x000202b0 GUI attribute queue management
UI_GetVMLibVersion -- Return library version string

Imports (Notable)

Beyond standard libc/libstdc++, libSharedLibrary64.so imports:

  • glob64 -- for scanning device paths
  • ioctl -- for physical device control
  • popen/pclose -- for shell command execution
  • system -- for system command execution (security concern)
  • readdir64/opendir/closedir -- directory enumeration
  • open64/lseek64/fseeko64/fopen64 -- large file support
  • inet_pton/inet_ntoa/htons -- network address conversion
  • wcstombs -- wide character conversion (for internationalization)

The use of system() and popen() is notable for security analysis.


Java Application Layer

Entry Point

KVMMain.main() parses command-line arguments:

args[0]: IP address
args[1]: Username
args[2]: Password
args[3]: KVM port
args[4]: Hostname
args[5]: VM port (default 623)
args[6]: Company ID
args[7]: Board ID
args[8]: Language

Static Initialization

On load:

  1. System.loadLibrary("iKVM64") -- KVM native library
  2. System.loadLibrary("SharedLibrary64") -- Virtual media native library

Key Java Classes

Class Package Role
KVMMain tw.com.aten.ikvm Entry point, argument parsing
KVMParam tw.com.aten.ikvm Configuration parameters
BoardInfo tw.com.aten.ikvm Board identification / feature flags
Viewer tw.com.aten.ikvm.ui Main window, thread management
RemoteVideo tw.com.aten.ikvm.ui Video display, input handling, JNI bridge
KeyMap tw.com.aten.ikvm.util Java VK → USB HID keycode translation
RMConnection tw.com.aten.ikvm.jni Connection JNI wrapper
RMDesktop tw.com.aten.ikvm.jni Desktop composition JNI wrapper
VirtualStorage tw.com.aten.vstorage Virtual media GUI + JNI bridge
ConnInfo tw.com.aten.bean Connection info data bean
UserInfo tw.com.aten.bean User credentials data bean
PlatformInfo tw.com.aten.bean Platform identification
VirtualUsbInfo tw.com.aten.bean Virtual USB configuration
InfoRepository tw.com.aten.bean Shared state repository

Threading Model

RemoteVideo manages three threads:

  1. CatchThread -- catchLoop() → native ProtocolHandler() loop
  2. DecodeThread -- runImage() → receive + decode video frames
  3. LazyWorker -- Periodic: refresh, updateImage, FPS calc

VirtualStorage manages per-device threads:

  • Dev1Thread / Dev2Thread / Dev3Thread -- Protocol loops for each VM device
  • EventThread -- Event processing
  • ShowMsgThread -- GUI message display
  • C_GUIObjectThreadThread -- GUI state updates
  • C_DevExistThread -- Device presence monitoring
  • GetDevHealthStatusThread -- Health status polling

Third-Party Code

  • ch.randelshofer.media.avi.AVIOutputStream -- AVI file writer for screen recording feature. Licensed separately (see license.html).

Class Hierarchy and Vtables

libiKVM64.so Vtable Addresses

All vtables are in .data section, accessible via _ZTV symbols:

Class Vtable Address RTTI Address Parent
TcpSocket 0x0022f740 0x0022f780 --
StreamSocket 0x0022f7c0 0x0022f7a0 TcpSocket
NtwStream (inline) -- StreamSocket
RMProtocol 0x00231020 0x00231080 --
RFBProtocol 0x0022f840 0x0022f8a0 RMProtocol
RMScreen 0x0022dd00 0x0022dd70 --
RFBScreen 0x0022f8c0 0x0022f940 RMScreen
RMKeyboard 0x0022f960 0x0022f9c0 --
RFBKeyboard 0x00230280 0x002302f0 RMKeyboard
RMMouse 0x0022dd80 0x0022dde0 --
RFBMouse 0x0022f9e0 0x0022fa40 RMMouse
RMPrivilege 0x00230f00 0x00230f40 --
RFBPrivilege 0x0022fcc0 0x0022fd00 RMPrivilege
RMCryto 0x00230f60 0x00230fa0 --
RFBKMCryto 0x0022f6a0 0x0022f6e0 RMCryto
RMDecoder 0x0022df80 0x0022dfb0 --
ASTVideoDecoder 0x0022de00 0x0022de30 RMDecoder
HermonVideoDecoder 0x0022de60 0x0022de90 RMDecoder
YarkonVideoDecoder 0x0022dec0 0x0022def0 RMDecoder
Pilot3VideoDecoder 0x0022df20 0x0022df50 RMDecoder
ASTDecode 0x00231100 0x00230ef0 --
ast2100 0x00230ea0 0x00230ed0 ASTDecode
ast_jpeg 0x00230fc0 0x00230ff0 ASTDecode
RMAuth 0x0022fa60 0x0022fa80 --
RMConnection 0x0022dfc0 0x0022e010 --
RMStorage 0x0022f800 0x0022f830 --
RMMisc 0x0022f700 0x0022f720 --
ErrMsg -- 0x0022df70 (exception type, 16 references)

RFBProtocol Object Layout

The RFBProtocol object (allocated size = 0x70 bytes) holds the protocol state.

Constructors:

  • RMProtocol::RMProtocol at 0x0011f9a0: Sets vtable to 0x00231030, sets mode_flags (0x20) = 0, operating_mode (0x30) = 0.
  • RFBProtocol::RFBProtocol at 0x00117cc0 (C1) / 0x00117c90 (C2): Calls RMProtocol::RMProtocol(), then sets rfb_major (0x40) = 3, rfb_minor (0x44) = 8, and vtable pointer to 0x0022f850.

The sub-object pointers (0x50-0x68) are NOT set by the RFBProtocol constructor. Instead, each sub-object's constructor stores its this pointer back into the RFBProtocol at the appropriate offset:

  • RFBMouse ctor stores this at protocol + 0x50
  • RFBKeyboard ctor stores this at protocol + 0x58
  • RFBScreen ctor stores this at protocol + 0x60
  • RFBPrivilege ctor stores this at protocol + 0x68

The NtwStream pointer at offset 0x08 is set by InitHandShake, which allocates a 0x628-byte NtwStream and stores it at this + 0x08.

Offset  Size  Type           Field               Notes
0x00    8     vtable*        vtable ptr           Points to 0x0022f850
0x08    8     NtwStream*     network stream       Buffered I/O layer (0x628 bytes)
0x10-0x1F     (inherited)    RMProtocol fields    (not explicitly used by RFB)
0x20    4     uint           mode_flags           Bit 0 = capture mode
0x24    4     (padding)
0x28    8     FILE*          capture_file         For replay/capture mode
0x30    4     int            operating_mode       0=normal, 1=replay, other=direct
0x34    4     (padding)
0x38    8     void*          source_data          File ptr or data ptr for replay
0x40    4     int            rfb_major            Always 3 (set by constructor)
0x44    4     int            rfb_minor            Always 8 (set by constructor)
0x48    1     uchar          fw_protocol_flag     Set by SetFWProtocol(); controls SetScreenUILang extra u32
0x49-0x4F     (padding)
0x50    8     RFBMouse*      mouse_handler        Set by RFBMouse constructor (0x28 bytes)
0x58    8     RFBKeyboard*   keyboard_handler     Set by RFBKeyboard constructor (0xB8 bytes)
0x60    8     RFBScreen*     screen_handler       Set by RFBScreen constructor (0x2080 bytes)
0x68    8     RFBPrivilege*  privilege_handler     Set by RFBPrivilege constructor (0x120 bytes)

RFBProtocol Vtable (at 0x0022f850)

Index  Offset  Address      Method
[0]    0x00    0x00117d60   ~RFBProtocol() (destructor, non-deleting)
[1]    0x08    0x00117dd0   ~RFBProtocol() (destructor, deleting)
[2]    0x10    0x00117fa0   InitHandShake(RMConnInfo_t)
[3]    0x18    0x00118190   Authenticate(Userinfo_t)
[4]    0x20    0x001183d0   ProtocolHandler()
[5]    0x28    0x00118230   ProcAlive(bool)
[6]    0x30    0x00118320   ProcSetScreenUI(int, int)
[7]    0x38    0x001183a0   ProcGetScreenUI()

RFBScreen Vtable (at 0x0022f8d0)

Verified from raw vtable bytes at 0x0022f8d0 (13 entries x 8 bytes).

Index  Offset  Address      Method
[0]    0x00    0x00118710   ~RFBScreen() (non-deleting, D1)
[1]    0x08    0x001187c0   ~RFBScreen() (deleting, D0)
[2]    0x10    0x00118900   SetPowerOnOff(uchar) [sends type 0x1A]
[3]    0x18    0x00118850   ScreenUpdate(ScreenReqInfo_t) [sends type 0x03]
[4]    0x20    0x00118960   ScreenCalibration() [sends type 0x17]
[5]    0x28    0x001189b0   ScreenSetPosition(ScreenPos_t) [sends type 0x15]
[6]    0x30    0x00118b80   ScreenSetInfo(ScreenInfo_t) [sends type 0x32]
[7]    0x38    0x00118c10   ScreenGetInfo() [copies screen info struct, no network I/O]
[8]    0x40    0x00118d50   ScreenSetFrameBuff(uchar*, uchar*)
[9]    0x48    0x00118d40   ScreenGetFreame() [returns framebuffer ptr at this+0x38]
[10]   0x50    0x00118da0   GeFrontGround()
[11]   0x58    0x00118cb0   GetScreenUILang() [handles server type 0x3C]
[12]   0x60    0x00118d60   GetCursorPos() [sends type 0x19] / MixedCursor

Correction from previous analysis: Index [2] is SetPowerOnOff (0x00118900), NOT ScreenUpdate. Index [3] is ScreenUpdate (0x00118850), NOT ScreenGetRect. Index [7] is ScreenGetInfo (0x00118c10), NOT SetPowerOnOff. The power control JNI functions call vtable offset 0x10 (index [2]) which is indeed SetPowerOnOff.

RFBMouse Vtable (at 0x0022f9f0)

Index  Offset  Address      Method
[0]    0x00    0x001194b0   ~RFBMouse() (non-deleting)
[1]    0x08    0x001194d0   ~RFBMouse() (deleting)
[2]    0x10    0x00119660   MouseAction(MouseInfo_t) [main entry: handles wheel repeat]
[3]    0x18    0x00119730   MouseSync() [sends type 0x07, value 0x0780]
[4]    0x20    0x00119780   MouseReset() [sends type 0x08 + mouse_type]
[5]    0x28    0x001197d0   MouseUpdateInfo() [sends type 0x37, clears config flag]
[6]    0x30    0x00119810   MouseGetPT() [returns 16 bytes from this+0x10]
[7]    0x38    0x00119820   MouseSetPT(PTerInfo_t) [sends type 0x36 + mode + type]
[8]    0x40    0x001198c0   ProcMouseInfo() [called by types 0x35, 0x37; reads 3 bytes]
[9]    0x48    0x00119910   MouseHotPlug() [sends type 0x3A, 1 byte only]

Note: Previous vtable mapping had SendMouse at index [2] and MouseAction at [3]. Corrected: index [2] is MouseAction (the JNI entry point which calls SendMouse internally), and the remaining methods follow the mangled symbol order. SendMouse at 0x001194f0 is called internally by MouseAction, not through vtable.

RFBMouse Object Layout (0x28 bytes)

Offset  Size  Type           Field              Notes
0x00    8     vtable*        vtable ptr         Points to 0x0022f9f0
0x08    8     RMProtocol*    protocol           Network stream access
0x10    1     byte           config_received    Set to 1 by ProcMouseInfo; cleared by MouseUpdateInfo
0x14    4     int            mouse_mode         From ProcMouseInfo byte 2; MouseReset sends it
0x18    4     int            encryption_enabled From ProcMouseInfo byte 1; SendMouse checks != 0
0x1c    4     int            additional_config  From ProcMouseInfo byte 3
0x20    8     RFBKMCryto*    crypto             AES encryption instance (0x2110 bytes)

Constructor (RFBMouse::RFBMouse at 0x001193F0): Allocates RFBKMCryto (0x2110 bytes) at offset 0x20. Initializes mouse_mode=1 (offset 0x14) and encryption_enabled=0 (offset 0x18 = unencrypted by default).

RFBKeyboard Vtable (at 0x00230290)

Index  Offset  Address      Method
[0]    0x00    0x0011a800   ~RFBKeyboard() (non-deleting)
[1]    0x08    0x0011a760   ~RFBKeyboard() (deleting)
[2]    0x10    0x0011a5d0   KeyboardAction(KeyInfo_t) [calls processVK via vtable[9]]
[3]    0x18    0x0011a5f0   KeyboardSync() [stub, returns 1]
[4]    0x20    0x0011a600   KeyboardReset()
[5]    0x28    0x0011a610   KeyboardUpdateInfo() [sends type 0x35]
[6]    0x30    0x0011a650   KeyboardGetType()
[7]    0x38    0x0011a660   KeyboardSetType(KeyTypeInfo_t)
[8]    0x40    0x0011a720   ProcKeyboardInfo() [called by type 0x35]
[9]    0x48    0x0011a670   Sendkey(int keysym, int down_flag) [always cleartext]
[10]   0x50    0x0011a920   processVK(KeyInfo_t) [VK -> X11 keysym via 3 maps]

Note: The KeyboardAction at vtable[2] calls vtable[9] (offset 0x48) which is processVK, not Sendkey. The decompiled code shows (**(code **)(*param_1 + 0x48))(param_1, ...) where 0x48 is vtable offset for processVK. After processVK returns the keysym, the caller invokes Sendkey (vtable[9] at offset 0x48) to transmit the event.

RFBPrivilege Vtable (at 0x0022fcd0)

Index  Offset  Address      Method
[0]    0x00    0x0011a0e0   ~RFBPrivilege() (non-deleting)
[1]    0x08    0x0011a100   ~RFBPrivilege() (deleting)
[2]    0x10    0x0011a120   ProcPrivilegeInfo() [called by type 0x39; reads 8+256 bytes]
[3]    0x18    0x0011a1b0   ExePrivilegeCtrl() [calls privilegeControl JNI helper]
[4]    0x20    0x0011a1f0   SendKickRequest(int, uchar*) [sends type 0x38, 73 bytes]
[5]    0x28    0x0011a1d0   ViewerConfig(int, uchar*) [stores session_id + config]

RFBPrivilege Object Layout (0x120 bytes)

Offset  Size  Type           Field              Notes
0x00    8     vtable*        vtable ptr         Points to 0x0022fcd0
0x08    8     RMProtocol*    protocol           Network stream access
0x10    4     int            session_word_lo    From ProcPrivilegeInfo
0x14    4     int            session_word_hi    From ProcPrivilegeInfo
0x18    256   u8[256]        aes_key_material   256 bytes read in ProcPrivilegeInfo
0x118   8     RFBKMCryto*    crypto             AES instance (0x2110 bytes, allocated in ctor)

Constructor (RFBPrivilege::RFBPrivilege at 0x00119fa0/0x0011a030):

  • Calls RMPrivilege::RMPrivilege(base) parent constructor
  • Stores this pointer at protocol+0x68 (so RFBProtocol can find it)
  • Allocates RFBKMCryto (0x2110 bytes) and stores at this+0x118
  • The SetCipherKey method on this crypto instance is a no-op (returns 1)

RMConnection Vtable (at 0x0022dfd0)

Index  Offset  Address      Method
[0]    0x00    0x00112870   ~RMConnection() (non-deleting)
[1]    0x08    0x001128b0   ~RMConnection() (deleting)
[2]    0x10    0x001128f0   ConnKeepActive()
[3]    0x18    0x001129e0   CheckVaildUser(Userinfo_t)
[4]    0x20    0x00112b40   QosControl()
[5]    0x28    0x00112b70   ConnEvenetHandle() [stub: returns 0]

RMDesktop Layout (Composition)

Offset  Size  Member
0x00    8     RFBKeyboard*
0x08    8     RFBMouse*
0x10    8     RFBScreen*
0x18    8     RFBPrivilege*
0x20    8     RFBProtocol* (parent)
...
0x30    --    (total allocated size = 0x30)

Security Analysis

Hardcoded Cryptographic Keys

AES-128-CBC key (libiKVM64.so, RFBKMCryto):

Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C
IV:  00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

This is the NIST AES test vector key. Used for mouse event encryption only (keyboard events are always cleartext). The key is hardcoded in the binary -- any client or MITM can decrypt.

RC4 key (libSharedLibrary64.so, plug-in packet encryption):

ASCII: BX80570E3110Q814A447
Hex:   42 58 38 30 35 37 30 45 33 31 31 30 51 38 31 34 41 34 34 37

Hardcoded 20-byte key at 0x00136202. Only encrypts the initial plug-in packet (credentials + USB descriptors).

RC4 key (libiKVM64.so, AST2050 video stream decryption):

ASCII: fedcba9876543210
Hex:   66 65 64 63 62 61 39 38 37 36 35 34 33 32 31 30

Hardcoded 16-byte key at 0x00230500. Used to optionally decrypt the AST2050 (encoding 0x57) video tile bitstream. Key is cyclically expanded to 256 bytes by Keys_Expansion() before RC4 KSA. Any client or MITM can decrypt.

Credential Handling

  • KVM authentication: Username (24 bytes) and password (24 bytes) are sent in plaintext over the TCP connection during the Authenticate phase. The server sends a 24-byte "challenge" which the client reads but discards -- there is no challenge-response mechanism. The checkValidUser JNI function at 0x00119C60 copies credentials from Java UserInfo fields using strcpy into fixed-size stack buffers (24 bytes for username, 96 bytes for password) -- potential buffer overflow if Java side provides longer strings.
  • Username and password are sent in plaintext (or RC4-encrypted with known key) in the VM plug-in packet
  • VM credentials are stored in global memory at fixed addresses:
    • Username: DAT_0023f848 (max 16 chars)
    • Password: DAT_0023f85c (max 20 chars)
  • KVM credentials are passed through JNI and stored in the UserInfo bean

Input Validation Concerns

  • sprintf used extensively (potential format string issues)
  • strcpy/strncpy used for string handling (buffer overflow risk)
  • system() and popen() imported by libSharedLibrary64.so (command injection risk)
  • No bounds checking on SCSI CDB parsing in some handlers
  • ISO file reading trusts values from ISO 9660 volume descriptor at offset 0x8050

Network Security

  • KVM connection: Modified RFB 3.8 over plain TCP (no TLS)
  • VM connection: Plain TCP with optional RC4 (known key)
  • HTTP upload: Plain HTTP with basic cookie auth (SID=)
  • No certificate validation, no HMAC, no replay protection
  • Keep-alive is unauthenticated

Software AES Implementation

The AES implementation in RFBKMCryto is a standalone software implementation (not using system OpenSSL/libcrypto). This means:

  • No hardware acceleration (AES-NI)
  • Potential for timing side-channels
  • Custom implementation may have subtle bugs

Key Data Structures and Globals

libiKVM64.so Global Variables

Symbol Address Type Purpose
connection BSS 0x1d2ce0 ptr RMConnection instance
desktop BSS 0x04823740 ptr RMDesktop instance
frameBuffer BSS 0x048236e8 ptr Direct ByteBuffer pointer
frameBufferLength BSS 0x048236e0 int Frame buffer size
cursorBuffer BSS 0x048236d0 ptr Cursor ByteBuffer pointer
cursorBufferLength BSS 0x048236c8 int Cursor buffer size
frameObj BSS 0x048236f0 jobject Java frame buffer reference
cursorObj BSS 0x048236d8 jobject Java cursor buffer reference
remoteVideo BSS jobject Java RemoteVideo reference
jvm BSS JavaVM* JVM pointer for callbacks
changeResolutionMid BSS 0x04823738 jmethodID Resolution change callback
errorHandlerMid BSS 0x04823730 jmethodID Error handler callback
genEventMid BSS 0x04823728 jmethodID Key event callback
privilegeCtrlMid BSS jmethodID Privilege control callback
addClipBoundsMid BSS 0x04823710 jmethodID Dirty rect callback
setViewerConfigMid BSS jmethodID Config callback
setNormalStartMid BSS jmethodID Normal start callback
getScreenUILangMid BSS 0x04823708 jmethodID OSD lang callback
getQuickCursorMid BSS 0x04823700 jmethodID Quick cursor callback
connInfoIpFid BSS 0x1d2cd8 jfieldID ConnInfo.ip field
connInfoPortFid BSS 0x1d2cd0 jfieldID ConnInfo.port field
g_config BSS 0x04823748 int Viewer config value
g_session_id DATA 0x001310cc int Session ID
buildtime DATA 0x001310d0 char[] Build timestamp
capsLock_status BSS 0x04823694 int CapsLock state
numLock_status BSS int NumLock state
scrollLock_status BSS int ScrollLock state
capsLock_LEDorig DATA 0x001310c4 int Original CapsLock LED
catchKey DATA 0x001310c8 int Catch key flag
filterFlag BSS 0x048236b0 int Input filter flag
flag BSS 0x048236bc int General flag
count BSS 0x048236c0 int Frame counter
captureFile BSS 0x04823628 ptr Screen capture file
captureName BSS 0x04823640 ptr Capture filename
ast_Buffer BSS 0x1d35c0 ptr AST decoder buffer
byTileYuv BSS 0x1d32c0 byte[] Tile YUV buffer
Cr_tab BSS 0x1d2aa0 int[] Cr→R lookup table
Cb_tab BSS 0x1d28a0 int[] Cb→B lookup table
Cr_Cb_green_tab BSS 0x1b28a0 int[] Cr/Cb→G lookup table
HTAC BSS 0x001311c0 struct AC Huffman table
HTDC BSS 0x00171320 struct DC Huffman table
HUFFMANCODE DATA 0x0012faa0 byte[] Huffman code constants
AC_CHROMINANCE_HUFFMANCODE DATA 0x00130320 byte[] AC chrominance table
AC_LUMINANCE_HUFFMANCODE DATA 0x001303e0 byte[] AC luminance table
DC_CHROMINANCE_HUFFMANCODE DATA 0x00130480 byte[] DC chrominance table
DC_LUMINANCE_HUFFMANCODE DATA 0x001304c0 byte[] DC luminance table
G_funct BSS 0x1d32b0 ptr Global function pointer
G_outBuf BSS 0x1d32a8 ptr Global output buffer
G_QT_TableSelection BSS 0x1d32a4 int Quantization table select
G_txb / G_tyb BSS 0x1d32bc/b8 int Tile position tracking

libSharedLibrary64.so Key Globals

See MOUNT_PROTOCOL.md for the java_vsui structure (0x0023f560) and per-device structure layout.

Symbol/Address Size Purpose
DAT_0023f560 (java_vsui) 0x2A0 Shared Java↔native configuration (see layout below)
st_VMMainInfo (0x0023f800) 0xD0 VM main info struct (cleared by InitVMSW)
DAT_0023f803 1 Encryption enabled flag (always 1 after JASWInit)
DAT_0023f804 1 SID authentication flag (always 1 after JASWInit)
DAT_0023f805-08 4 SID bytes (4 bytes from rand() seeded with time())
DAT_0023f810 8 Per-device structure base pointer (from VMInfoCalloc)
DAT_0023f818 8 File-backed storage function table pointer
DAT_0023f820 8 Socket function pointer table (indexed by conn type)
DAT_0023f830 8 Physical storage function table pointer
DAT_0023f848 16 Username string (max 16 chars, from StoreVMInfoFromGUI)
DAT_0023f85c 20 Password string (max 20 chars, from StoreVMInfoFromGUI)
DAT_0023f874 ~72 BMC IP address string (IPv6 bracket-stripped)
DAT_0023f8bc 4 Connection type (0=IPv4, 1=IPv6, from DetermineTCPType)
DAT_0023f8c0 4 Descriptor mode flag (always 1 after SetupVMInfoBetSWAndFW)
DAT_0023f8c4 4 VM device count
vuDevRespData (0x0023e240) 824 Hardcoded USB descriptor table (8 entries at 0x67 stride)
st_VSDevConfigDescriptor (0x0023e580) 540 Device config descriptor table
st_VSIADDescriptor (0x0023e7a0) 60 IAD descriptor table (5 entries)

java_vsui structure layout (0x0023f560, 0x2A0 bytes):

Offset Size Field Set by
0x000 20 Username (temp) JASetUNamePwdIPPort (via strcpy, unbounded)
0x014 20 Password (temp) JASetUNamePwdIPPort (via strcpy, unbounded)
0x02C 72 IP address JASetUNamePwdIPPort (via memcpy, length-bounded)
0x074 512 Image file path VUSPlugIn
0x294 4 Platform version JASWInit
0x298 1 Company ID JASWInit
0x299 1 Board ID JASWInit

Per-device structure (allocated at DAT_0023f810, stride = 0x15920):

Slots: devCount + 1 (extra slot for temporary status query connections).


Round 5 Verification Results

The following items were independently re-verified by decompiling all relevant functions from Ghidra. All decompilations were performed on the binary /liblinux_x86_64__V1.0.5/libiKVM64.so-730126.

ProtocolHandler Dispatch Table (VERIFIED)

The jump table at 0x00127790 was read as raw bytes and decoded. It contains exactly 61 entries (types 0x00 through 0x3C), each a 32-bit signed offset relative to the table base address. The default handler is at 0x001183f5 (returns the type byte to the caller without consuming more data).

The 7 non-default handlers are confirmed at these exact target addresses:

Type Target Handler
0x00 0x0011849a Calls RFBScreen::ScreenDecode via vtable[0x58]
0x04 0x001184de Calls RFBScreen::ScreenCursorPosProc via vtable
0x16 0x00118488 Inline: StreamRead8() (reads 1 status byte)
0x35 0x0011844d Calls RFBKeyboard::ProcKeyboardInfo at offset 0x40, then falls through to 0x37 handler
0x37 0x00118457 Calls RFBMouse::ProcMouseInfo at offset 0x40
0x39 0x00118479 Calls RFBPrivilege::ProcPrivilegeInfo at vtable[0x10]
0x3C 0x0011846a Calls RFBScreen::GetScreenUILang via vtable[0x58]

The raw jump table hex at 0x00127790 (first 7 non-default offsets, all others are the default 0xFFFF0C65 = -62363):

Type 0x00: 0xFFFF0D0A -> 0x0011849A
Type 0x04: 0xFFFF0D4E -> 0x001184DE
Type 0x16: 0xFFFF0CF8 -> 0x00118488
Type 0x35: 0xFFFF0CBD -> 0x0011844D
Type 0x37: 0xFFFF0CC7 -> 0x00118457
Type 0x39: 0xFFFF0CE9 -> 0x00118479
Type 0x3C: 0xFFFF0CDA -> 0x0011846A

Confirmed: Type 0x04 IS cursor position (not screen control). The handler calls ScreenCursorPosProc which reads cursor_x, cursor_y, cursor_width, cursor_height, cursor_type, and optionally cursor pixel data. Previous documentation was correct on this point.

Authentication Flow (VERIFIED)

RFBProtocol::Authenticate at 0x00118190 decompilation confirms:

  1. StreamRead(auStack_28, 0x18) -- reads 24-byte challenge into LOCAL variable
  2. StreamWrite(&stack0x00000008, 0x18) -- writes 24 bytes from username parameter
  3. StreamWrite(&stack0x00000020, 0x18) -- writes 24 bytes from password parameter
  4. StreamRead32() -- reads auth_result (u32)

The challenge is read into auStack_28[32] (local stack variable) and NEVER referenced again. It is genuinely discarded. The username and password parameters come from the Userinfo_t struct passed by value on the stack.

The checkValidUser JNI at 0x00119c60 copies username with strcpy(local_a8, __src) into a 24-byte buffer and password with strcpy(local_90, __src_00) into a 96-byte buffer. This is a buffer overflow vulnerability if the Java side provides strings longer than 24/96 bytes respectively.

Security finding confirmed: Credentials are sent in plaintext. The 24-byte challenge is theater -- it exists in the protocol but is completely ignored by the client implementation.

Connection Handshake (VERIFIED)

All handshake functions were decompiled. Key findings:

  • InitHandShake (0x00117fa0): Allocates NtwStream (0x628 bytes), stores at this+0x08. Calls Connect(), ProcVersion(), ProcSecurity(). Returns 0 on any failure, 1 on success.

  • ProcVersion (0x00117e40): Reads 12 bytes, parses with sscanf("RFB %03d.%03d\n"), compares against this+0x40 (=3) and this+0x44 (=8). Echoes the same 12 bytes back.

  • ProcSecurity (0x00117f10): Reads count (u8), then reads count security types (u8 each) in a loop. The LAST type read is selected and sent back (1 byte). Note: the variable holding the selected type (unaff_R13B) is the register that holds the last value from the loop -- confirming "last type wins" behavior.

  • Authenticate (0x00118190): See above.

  • ProcClientInit (0x00117ee0): Sends single byte 0x00 (shared_flag = exclusive).

  • ProcServerInit (0x00118040): Reads standard RFB ServerInit (2+2+16+4+name bytes), all discarded. Then reads 4 bytes padding (skip), session_id (u32), and 4 bytes of aes_key_seed (read as 4 individual StreamRead8 calls). Creates a temporary RFBPrivilege (0x120 bytes), calls ViewerConfig(session_id, &key_seed) on it via vtable[0x28], then immediately destroys it via vtable[0x00].

GetDecoder (VERIFIED)

RMDecoder::GetDecoder at 0x00112560 is a singleton factory. The global DecoderHandle is checked first -- if non-NULL, it is returned immediately. Otherwise a switch on the encoding type:

  • 0x57 or 0x58: ASTVideoDecoder (0x58 bytes)
  • 0x59: HermonVideoDecoder (0x58 bytes)
  • 0x60: YarkonVideoDecoder (0x60 bytes)
  • 0x61: Pilot3VideoDecoder (0x458 bytes)

There is NO default case and no fallback. Unknown encoding types result in DecoderHandle remaining NULL, and the function returns NULL.

JNI Function Behavior (VERIFIED)

JNI Function Behavior
changeLEDstate (0x00120cd0) Purely local X11. Opens display, simulates key press/release for CapsLock (VK 0x39->0xFFE5), ScrollLock (VK 0x47->0xFF14), or NumLock (VK 0x53->0xFF7F) using XTestFakeKeyEvent, closes display. NO network I/O.
updateInfo (0x00120be0) Calls MouseUpdateInfo() via vtable on desktop+0x08 (RFBMouse), vtable offset 0x28. Sends type 0x37 on wire.
changeScreenInfo (0x00120c00) Calls ScreenGetRect() then ScreenSetInfo() via vtable on desktop+0x10 (RFBScreen). Sends type 0x32 if resolution changed. Also refreshes JNI buffer references.
doCatch (0x00121570) Sets global filterFlag = param_3. No message processing.
catchLoop (0x001216c0) Empty no-op function (contains only return).
refresh (0x00121510) JNI MonitorEnter/MonitorExit on frameObj, resets flag. No network I/O.
sendPrivilegeCtrl (0x00121970) Extracts 64-byte array from Java via GetByteArrayRegion(0, 0x40), calls SendKickRequest(action, data) via vtable on desktop+0x18 (RFBPrivilege) offset 0x20. Sends type 0x38 (73 bytes).

Round 6 Final Sweep Results

The following items were verified and discovered by decompiling all relevant functions from Ghidra. All decompilations were performed on the binary /liblinux_x86_64__V1.0.5/libSharedLibrary64.so-161223.

Byte-Level Wire Format Verification (ALL CONFIRMED CORRECT)

Every critical packet format was verified by decompiling the function that constructs it. All documented byte offsets in MOUNT_PROTOCOL.md are accurate:

  • PlugIn (FillUSBPlugInPkt at 0x001167c0): 52 bytes + USB descriptors
  • SetEP (FillSetEPCMDPkt at 0x00116234): 16 bytes, 3 hardcoded endpoints
  • KeepAlive (FillKeepAlivePkt at 0x0011b1c2): 12 bytes, payload 0xFFFFFFFF
  • DevStatus (Core_GetDevStatusFromFW at 0x0011741e): 8-byte query, type=0x08
  • HTTP Port (GetHttpPortFromFW at 0x0011cdd4): 8-byte query, type=0x0A (leaks socket)
  • UnMount (Core_VM_SendUnMountPkt at 0x0011bdf2): 8 bytes, type=5
  • PDU Header (State_Fn_RX_PDU_TAG at 0x0011b24c): type=BE, length=LE
  • CSW (FillCSW at 0x00119fe8): 21 bytes, both IN and OUT paths verified
  • Data PDU (FillDataPDUTag at 0x00119d10): 8-byte header, IN direction only

Key Discoveries

  1. Agent 4 SID/USB ID correction: Offsets 0x2C-0x2F in PlugIn packet are SID bytes (from GetRandomTimeStamp), NOT USB vendor/product IDs as Agent 4 incorrectly labeled. MOUNT_PROTOCOL.md correctly identifies these as SID bytes.

  2. VMInfoCalloc extra slot: calloc((devCount+1), 0x15920) allocates an extra slot beyond the device count, used as a temporary connection slot for status queries by Core_GetDevStatusFromFW.

  3. SetupVMInfoBetSWAndFW descriptor reset: Before mount, all HID keyboard/ mouse descriptors and all 5 IAD descriptors are explicitly disabled. Only the current device's descriptor is re-enabled by MtMethod_Media.

  4. StoreVMInfoFromGUI validation: Username < 17 chars, password < 21 chars. IPv6 addresses have brackets stripped. However, JASetUNamePwdIPPort copies credentials via unbounded strcpy BEFORE this validation occurs.

  5. Floppy WRITE(10) 32-bit limitation: Uses fseek (32-bit) while READ(10) uses fseeko64 (64-bit). Latent bug for images > 2GB.

  6. FillErrorData logic: CSW status is PASS (0x00) if both ASC and ASCQ are zero; otherwise FAIL (0x01). No finer-grained status mapping.

  7. SCSI handlers fully enumerated: ISO has 26 opcodes, floppy has 10. Newly documented handlers: MECHANISM STATUS (0xBD), START STOP UNIT (0x1B), SET READ AHEAD (0xA7), SET CD SPEED (0xBB), READ BUFFER (0x3C).

New Function Addresses

Address Function Purpose
0x001178c8 InitVMSW Clear st_VMMainInfo, init queues, generate SID
0x00120bb2 VMInfoCalloc calloc((devCount+1), 0x15920)
0x00117b92 SetState_ReadPDU Reset state to 0x01, expect 8-byte header
0x0011ced4 SetupVMInfoBetSWAndFW Pre-mount setup with verified address
0x0011f958 UI_Mount_VM Full mount orchestration
0x0011d28c StoreVMInfoFromGUI Credential validation and storage
0x0011667c UI_PrepareVMResource Resource preparation and memory allocation
0x0011b24c State_Fn_RX_PDU_TAG PDU header parser and dispatch
0x0011ac68 State_Fn_RX_CBW CBW parsing state
0x00117cc8 State_Fn_RX_BULK_OUT_DATA Bulk OUT data receive state
0x0011b694 State_Fn_RX_MOUNT_DEV_STATUS Mount status response handler
0x001288be MachanismStatus MECHANISM STATUS (0xBD) handler
0x00128aa0 Startstopunit START STOP UNIT (0x1B) handler
0x00128378 SetReadAhead SET READ AHEAD (0xA7) handler
0x00128e80 SetCDSpeed SET CD SPEED (0xBB) handler
0x0012a7aa ReadBuffer READ BUFFER (0x3C) handler

Areas for Further Investigation

Completed (Final Round)

  1. [DONE] ScreenDecode exact wire format -- RFBScreen::ScreenDecode at 0x00119090 fully decompiled. Exact StreamRead sequence after type 0x00: StreamReadSkip(3), StreamRead16 x4 (x, y, w, h), StreamRead32 (encoding), StreamRead32 (frame_number), StreamRead32 (data_length), StreamRead(buf, data_length). Total: 23 bytes header + data_length. The 3-byte skip covers standard RFB padding(1) + num_rects(2). Width/height are signed 16-bit and abs()'d before decode. See KVM_PROTOCOL.md for complete byte-level docs.

  2. [DONE] Complete client-to-server message enumeration -- All 22 functions containing StreamWrite8 calls were identified via code search. 19 unique client-to-server message type bytes found (one StreamWrite8 is ProcClientInit which writes 0x00 as shared_flag, not a message type; two StreamWrite8 calls are for encrypt_flag and button_mask subfields). The previously missing type is 0x35 (KeyboardUpdateInfo) -- a 1-byte client request for keyboard+mouse config, sent by RFBKeyboard::KeyboardUpdateInfo at 0x0011a610. Complete list: 0x03, 0x04, 0x05, 0x07, 0x08, 0x15, 0x16, 0x17, 0x19, 0x1A, 0x32, 0x35, 0x36, 0x37, 0x38, 0x3A, 0x3B, 0x3C, 0x3D.

  3. [DONE] KeepAlive (type 0x16) details -- Server sends type 0x16 + 1 byte (status). The ProtocolHandler inline handler at 0x00118488 calls StreamRead8() and discards the result (not stored or checked). The status byte has no significance. Client ACK: RFBProtocol::ProcAlive(bool) at 0x00118230 sends type 0x16 + the bool parameter. The sendKeepAliveAck JNI at 0x00121a00 always passes 1 (true) via vtable offset 0x28. A client should always send 0x16 0x01 in response.

  4. [DONE] Object allocation sizes -- All sizes verified from operator_new calls in constructors: NtwStream=0x628, RFBProtocol=0x70, RFBScreen=0x2080, RFBMouse=0x28, RFBKeyboard=0xB8, RFBPrivilege=0x120, RFBKMCryto=0x2110, RMConnection=0x10, RMDesktop=0x30, TcpSocket=0x60, ErrMsg=0x44. The RMDesktop constructor (0x0011e0f0) allocates all sub-objects: RFBKeyboard (0xB8), RFBMouse (0x28), RFBScreen (0x2080), RFBPrivilege (0x120).

  5. [DONE] X11 indicator state bit mapping correction -- The init JNI at 0x0011fbd0 reads XkbGetIndicatorState and tests: bit 0 (& 1) = scrollLock, bit 1 (& 2) = numLock, bit 2 (& 4) = capsLock. Previous documentation had these reversed (bit 0 = CapsLock). Corrected in KVM_PROTOCOL.md.

High Priority (Previously Completed)

  1. [DONE] RFBProtocol::ProtocolHandler dispatch table -- Fully mapped. Jump table at 0x00127790 has 61 entries (types 0x00-0x3C). Only 7 have non-default handlers: 0x00 (FramebufferUpdate), 0x04 (CursorPosition), 0x16 (KeepAlive), 0x35 (KeyboardInfo+MouseInfo), 0x37 (MouseInfo only), 0x39 (PrivilegeInfo), 0x3C (GetScreenUILang). All handler code is inline within the ProtocolHandler function body. Previous documentation had incorrect type IDs (0x31, 0x33 don't exist; 0x38 is client-only).

  2. [DONE] Video decoder internals (ast_jpeg / 0x58) -- The AST2100+ decoder (ast_jpeg class) uses a JPEG-like scheme with 4:2:0 YCbCr subsampling: 4 Y blocks + 1 Cb + 1 Cr per tile (16x16 pixels). Fixed Huffman tables (not embedded in stream). Tile command stream uses 2-bit command types: same-position or new-position, with two QT table selections. 16-byte header provides quantization levels, tile sizes, and total size. See KVM_PROTOCOL.md for full wire format.

2b. [DONE] AST2050 decoder (ast2100 class, encoding 0x57) -- Fully decompiled. Uses a hybrid VQ + Huffman + IDCT decoder with optional RC4 decryption. RC4 key is the hardcoded string "fedcba9876543210" (at 0x00230500). The 4-byte header provides mode, quality, and chip revision (0x01A6 = 16x16 tiles, 0x01BC = 8x8 tiles). Tile commands are encoded as 4-bit nibble codes in the bitstream. VQ uses a 4-entry COLOR_CACHE palette. See KVM_PROTOCOL.md for complete wire format documentation.

  1. [DONE] system() and popen() call sites -- system() PLT stub at 0x001143b0, popen() PLT stub at 0x00112ae0. Both are called from the FAT filesystem emulation (Linux_TFATFileSystemImage_*) and physical device scanning (Linux_ScanPhyStor_*) code paths. No system()/popen() calls were found in the main protocol engine, SCSI handlers, or mount path. The calls appear limited to folder mount file operations and hardware enumeration. Security risk is mitigated by the fact that user-controlled file paths do not directly reach these calls (they go through StoreVMInfoFromGUI validation first), but the folder mount path should still be considered a potential command injection surface.

  2. [DONE] Authentication flow -- RFBProtocol::Authenticate at 0x00118190 uses ATEN-proprietary plaintext auth: reads 24-byte challenge (discarded), sends 24-byte username + 24-byte password in cleartext, reads 4-byte result. No VNC DES challenge-response or other standard security type handling. The security type negotiation is a formality -- the client blindly selects the last type offered.

Medium Priority

  1. Physical device scanning -- The Linux_ScanPhyStor_* functions enumerate hardware devices on the client machine. What paths are scanned? Are there symlink/TOCTOU issues?

  2. FAT filesystem emulator -- The TFATFileSystem_* family implements a complete FAT16 filesystem in memory for folder mounting. The complexity here is a potential source of bugs (path traversal, buffer overflows in filename handling, etc.).

  3. [DONE] Error message / exception flow -- The ErrMsg class is a 0x44-byte struct with error_code (int) + error_message (char[64]). Error codes: -1 (socket closed/failed), -2 (timeout), -3 (multi-session not supported), -4 (file closed), -5 (file corrupted). Exceptions are thrown in TcpSocket read/write and video decoders, caught in runImage/checkValidUser/keepActive JNI functions. The errorHandler(int) JNI callback (registered at init) propagates error codes to Java. Socket timeout is 30 seconds.

  4. [DONE] Hermon and Yarkon decoders -- Fully decompiled. Hermon uses raw 16bpp/8bpp tiles with simple pixel format conversion. Yarkon uses Huffman/RLE compression with RFB Hextile sub-encoding for incremental updates. Both throw "Not Support Multi-Session!" for encoding 0x10001. See KVM_PROTOCOL.md for complete wire formats.

  5. [DONE] Screen recording subsystem -- The screenRecording* exports use a simple file-based capture/playback mechanism. startCapture opens a file and sets a flag; the ProtocolHandler's type 0x00 handler writes received frame data to the capture file when the flag is set. Playback reads from a file via TcpSocket's file mode (offset 0x44 != 0). Recording is raw protocol data (not AVI). The Java side uses AVIOutputStream for the actual AVI recording. Huffman_Compress/Uncompress are used by the Yarkon decoder, not directly by the recording subsystem.

Low Priority

  1. [DONE] Keyboard LED synchronization -- The keyboardAction JNI function at 0x00121000 opens/closes an X11 display connection for each lock key event. It reads the indicator state via XkbGetIndicatorState(display, 0x100, &state): bit 0 = ScrollLock, bit 1 = NumLock, bit 2 = CapsLock. If the lock is not active, the VK code is modified with 0xFF in the low byte. The X11 connection is ephemeral (opened, read, closed) -- no persistent display handle.

  2. IPv6 support -- The Linux_VMTCPConnect_IPv6_Default function and bracket/scope stripping logic in the Java side. IPv6 parsing is notoriously error-prone.

  3. HTTP upload protocol -- The multipart/form-data construction and response parsing in MtMethod_WebISO / MtMethod_UploadIMA. Cookie handling (SID=) may have injection issues.

  4. [DONE] USB descriptor assembly -- FillUSBPlugInPkt at 0x001167c0 fully decompiled and verified byte-by-byte. The descriptor mode flag (DAT_0023f8c0, always 1 after SetupVMInfoBetSWAndFW) controls idProduct (0x2222 vs 0x1111). 8 entries from vuDevRespData table are assembled, with Entry 1 (Configuration Descriptor) dynamically including Interface + Endpoint descriptors. Before mount, all HID/IAD descriptors are disabled and only the current device's Mass Storage interface is enabled.

Remaining Open Questions (from final round)

  • Server-side KeepAlive status byte: The server sends a status byte with each KeepAlive (type 0x16), but the client discards it. Does the BMC send different values (e.g., 0 for idle, 1 for active)? Only observable via packet capture against a live BMC.

  • Type 0x35 dual-purpose: The type 0x35 is used bidirectionally -- the client sends it as a 1-byte config request, the server sends it as a 6-byte config message. This asymmetry is unusual. Verify with packet captures whether the BMC correctly distinguishes the two directions by context.

  • frame_number semantics: The first frame has frame_number=1 (triggers resolution change notification), subsequent frames have frame_number=0. Are there other frame_number values? Does it ever increment beyond 1? Only observable from live traffic.

  • PrivilegeInfo session_word meaning: The (lo=1, hi=4) comparison in ProcPrivilegeInfo triggers SetThreadNormaleStart(0). What does this mean? Is (1,4) a "session takeover" indicator? What are other observed values?

  • Server behavior on unknown client types: What happens if the client sends a message type the BMC does not expect? Does the BMC disconnect, ignore, or desynchronize?

  • Nagle's algorithm: The enableNagles() function in libiKVM64.so is a no-op stub. Similarly, libSharedLibrary64.so never sets TCP_NODELAY -- send() calls use flags=0. This means Nagle's algorithm is active on both the KVM and VM connections, which could cause latency issues for small messages. A real client implementation should consider setting TCP_NODELAY.

  • SIGPIPE handling (VM library): libSharedLibrary64.so does not set MSG_NOSIGNAL on send() and has no signal(SIGPIPE, SIG_IGN) handler (unlike libiKVM64.so which has BrokenPipe_handle). A broken pipe on the VM connection would deliver SIGPIPE and could crash the JVM.

Additional Discovered Functions and Addresses

Authentication path:

Function Address Purpose
RFBProtocol::InitHandShake 0x00117fa0 TCP connect + version + security negotiation
RFBProtocol::Authenticate 0x00118190 ATEN plaintext auth + ClientInit + ServerInit
RFBProtocol::ProcVersion 0x00117e40 RFB version exchange
RFBProtocol::ProcSecurity 0x00117f10 Security type negotiation
RFBProtocol::ProcClientInit 0x00117ee0 Send shared_flag = 0x00
RFBProtocol::ProcServerInit 0x00118040 Read server init + ATEN extensions
RFBProtocol::SetFWProtocol 0x00118530 Set firmware protocol flag byte

Server message handlers (called from ProtocolHandler):

Function Address Server Type Purpose
RFBScreen::ScreenDecode 0x0010E7E8 (DEFAULT) / 0x00119090 (IMPORTED) 0x00 Video frame decode
RFBScreen::ScreenCursorPosProc 0x0010E7F8 (DEFAULT) / 0x00118FB0 (IMPORTED) 0x04 Cursor position + image
RFBKeyboard::ProcKeyboardInfo 0x0011A720 0x35 Keyboard mode/type config
RFBMouse::ProcMouseInfo 0x001198C0 0x35, 0x37 Mouse mode/type/encrypt config
RFBPrivilege::ProcPrivilegeInfo 0x0011A120 0x39 Session + AES key material
RFBPrivilege::ExePrivilegeCtrl 0x0011A1B0 -- Forward privilege data to Java
RFBPrivilege::ViewerConfig 0x0011A1D0 -- Store session_id + config
RFBScreen::GetScreenUILang 0x00118CB0 0x3C OSD language response

Client message senders:

Function Address Client Type Purpose
RFBScreen::ScreenUpdate 0x00118850 0x03 Send FramebufferUpdateRequest (10 bytes)
RFBKeyboard::Sendkey 0x0011A670 0x04 Send KeyEvent (18 bytes, always cleartext)
RFBMouse::SendMouse 0x001194F0 0x05 Send PointerEvent (encrypted or clear)
RFBMouse::MouseSync 0x00119730 0x07 Send MouseSync (value 0x0780)
RFBMouse::MouseReset 0x00119780 0x08 Send MouseReset (+ mouse_type)
RFBScreen::ScreenSetPosition 0x001189B0 0x15 Send screen position
RFBProtocol::ProcAlive 0x00118230 0x16 Send KeepAlive ACK
RFBScreen::ScreenCalibration 0x00118960 0x17 Trigger screen recalibration
RFBScreen::GetCursorPos 0x00118D60 0x19 Request cursor position from server
RFBScreen::SetPowerOnOff 0x00118900 0x1A Power control (off/on/reset/soft-off)
RFBScreen::ScreenSetInfo 0x00118B80 0x32 Send resolution
RFBKeyboard::KeyboardUpdateInfo 0x0011A610 0x35 Request keyboard+mouse config (1 byte)
RFBMouse::MouseSetPT 0x00119820 0x36 Set pointer type (mode + type)
RFBMouse::MouseUpdateInfo 0x001197D0 0x37 Request mouse config from server
RFBPrivilege::SendKickRequest 0x0011A1F0 0x38 Send kick request (73 bytes)
RFBMouse::MouseHotPlug 0x00119910 0x3A Mouse hot-plug reset (1 byte)
RFBProtocol::ProcQos 0x00118290 0x3B Send QoS parameters (3 x u32)
RFBProtocol::ProcGetScreenUI 0x001183A0 0x3C Request OSD language (1 byte)
RFBProtocol::ProcSetScreenUI 0x00118320 0x3D Set OSD language (conditional u32)

JNI helper functions:

Function Address Purpose
sendViewerConfig 0x0011FB40 Send g_config + g_session_id to Java
storeViewerConfig 0x001216A0 Store session_id and config bytes (strncpy 4 bytes)
SetThreadNormaleStart 0x00121640 Notify Java of normal/non-normal session
privilegeControl 0x00121580 Forward privilege info (lo, hi, 256-byte key) to Java
getScreenUILangConfig 0x00121B80 Forward OSD lang (2 x u32) to Java
addClipBounds 0x0011FAA0 Forward dirty rectangles (4 x int) to Java
getQuickCursor (called from ScreenCursorPosProc) Notify Java of cursor update

Error handling functions:

Function Address Purpose
ErrMsg typeinfo 0x0022DF70 C++ exception type (0x44 bytes: int code + char[64] msg)
ErrMsg typeinfo-name 0x00122415 "6ErrMsg" mangled name
TcpSocket::read 0x00117310 Throws ErrMsg on socket errors (-1, -2, -4, -5)
TcpSocket::write 0x00117530 Throws ErrMsg on socket errors (-1, -2)

NtwStream Implementation Details

The NtwStream class provides buffered network I/O with mutex-based write batching. The object layout includes a pthread_mutex_t at the beginning, a 1518-byte internal write buffer, and pointers to the underlying socket.

Object layout (partial):

Offset  Size   Description
0x00    varies pthread_mutex_t (write lock, ~40 bytes on Linux x86-64)
0x28    0x5F0  Internal write buffer (1520 bytes = offset 0x618 - 0x28)
0x618   8      Write cursor pointer (points into buffer at 0x28..0x618)
0x620   8      Socket vtable pointer (StreamSocket/TcpSocket instance)

Buffer size: The internal write buffer spans from offset 0x28 to 0x618 (1520 bytes, or 0x5F0). This is based on the capacity check in StreamWrite(): if ((this - __dest) + 0x616 < param_2) where __dest = *(this + 0x618). This equates to an Ethernet MTU-sized buffer (1518 bytes is standard Ethernet MTU + 2 bytes for alignment).

StreamWriteStart (0x0010f370): Simply calls pthread_mutex_lock() on the mutex at the beginning of the NtwStream object. This locks the stream for an atomic write batch.

StreamWriteFlush (0x0010f380): Sends buffered data and releases the lock:

  1. If write cursor != buffer start (i.e., there is buffered data):
    • Calls the socket's write virtual method (vtable offset 0x20) with the buffer contents from 0x28 through the cursor position
    • Resets write cursor to buffer start (0x28)
  2. Calls LeaveCriticalSection() which wraps pthread_mutex_unlock()

StreamWrite (0x0010f1f0): Buffers data or flushes if buffer is full:

  • If the data fits in the remaining buffer space: memcpy() into buffer and advance the cursor
  • If data exceeds remaining space:
    1. Flush current buffer directly via socket write
    2. Send the new data directly via socket write (bypasses buffer)
    3. Reset cursor to buffer start

StreamRead (0x0010f0a0): Reads exactly param_2 bytes from the socket:

  • Loops calling the socket's read virtual method (vtable offset 0x18)
  • Accumulates partial reads until all bytes are received or an error occurs
  • Returns total bytes read (should equal param_2 on success)

StreamReadSkip (0x0010f180): Allocates a temporary buffer, reads N bytes into it via StreamRead(), then frees the buffer. Simple but wasteful -- it allocates and zeroes memory just to discard the bytes.

StreamWriteSkip (0x0010f3f0): Same pattern as StreamReadSkip -- allocates a zeroed buffer and writes it via StreamWrite(). This sends N zero bytes.

StreamRead8/16/32: Thin wrappers around StreamRead:

  • StreamRead8(): Reads 1 byte, returns it directly
  • StreamRead16(): Reads 2 bytes, returns big-endian u16
  • StreamRead32(): Reads 4 bytes, returns big-endian u32

StreamWrite8/16/32: Thin wrappers around StreamWrite:

  • StreamWrite8(val): Writes 1 byte
  • StreamWrite16(val): Writes 2 bytes big-endian (high byte first)
  • StreamWrite32(val): Writes 4 bytes big-endian (MSB first)

Socket layer (TcpSocket):

  • TcpSocket::read (0x00117310): Uses select() with 30-second timeout + recv(). Throws ErrMsg exceptions: -1 (closed), -1 (failed), -2 (timeout). Also supports file replay mode (offset 0x44 != 0) using fread().
  • TcpSocket::write (0x00117530): Uses select() with 30-second timeout + send(). Throws ErrMsg exceptions: -1 (closed), -1 (failed), -2 (timeout).
  • TcpSocket::EstablishConnection (0x00117770): Uses getaddrinfo() + socket() + connect() for client mode; bind() + listen() + accept() for server mode (offset 0x5c != 0). Calls enableNagles() which is a no-op stub (Nagle's algorithm is not disabled -- TCP_NODELAY is never set).
  • TcpSocket::EndSock (0x00117aa0): Calls shutdown(fd, SHUT_RDWR) then close(fd). Manages a reference count (RefSocketCount) for shared listen socket cleanup.

Connection Teardown

destory [sic] (Java_tw_com_aten_ikvm_ui_RemoteVideo_destory at 0x0011ffd0):

The destory() JNI method performs complete session cleanup:

  1. Saves desktop and connection global pointers to local variables
  2. Sets both global pointers to NULL immediately (prevents double-free)
  3. If desktop was non-NULL:
    • Calls RMDesktop::~RMDesktop() which destroys the composition:
      • Destroys the RFBScreen (includes video decoder cleanup)
      • The destructor chain destroys keyboard, mouse, screen, privilege objects
    • Frees the RMDesktop memory via operator delete
  4. If connection was non-NULL:
    • Calls the deleting destructor via vtable (offset 0x08) which:
      • Destroys the RFBProtocol (including NtwStream)
      • NtwStream destructor destroys the TcpSocket
      • TcpSocket destructor calls EndSock(): a. shutdown(socket_fd, SHUT_RDWR) -- half-close both directions b. close(socket_fd) -- release file descriptor c. Decrements RefSocketCount; if zero, also shuts down Listenfd
      • Frees the TcpSocket memory

There is no graceful disconnect message. The connection is torn down by simply shutting down the TCP socket. There is no RFB "disconnect" or "bye" message sent to the server. The server detects the disconnect via TCP RST or FIN when the socket is closed.

Server disconnect handling: When the server disconnects, the next TcpSocket::read() call will either:

  • Return 0 (EOF) -> throws ErrMsg(-1, "Read: Socket closed")
  • Return -1 (error) -> throws ErrMsg(-1, "Socket Read Failed")
  • Timeout after 30 seconds -> throws ErrMsg(-2, "Socket Read Timeout")

The C++ exception propagates up through the JNI call stack. The Java-side catch handler in runImage() / catchLoop() receives the error code via the errorHandler(int) callback and terminates the session.

AST2050 Decoder Internals (ast2100 class)

Function addresses:

Function Address Purpose
ast2100::ast2100 (C1) 0x0011b5f0 Constructor
ast2100::ast2100 (C2) 0x0011b5c0 Constructor (delegating)
ast2100::~ast2100 (D1) 0x0011ca10 Destructor
ast2100::~ast2100 (D2) 0x0011c9e0 Destructor (delegating)
ast2100::PsudoStreamSwap16 0x0011b620 Byte-swap u16 (big-endian)
ast2100::GetINFData 0x0011b640 Initialize decoder info defaults
ast2100::InitParameter 0x0011b6f0 Set tile/mode params from chip revision
ast2100::SetBuffer 0x0011b7e0 Set framebuffer dimensions and mode
ast2100::Keys_Expansion 0x0011b830 Expand RC4 key to 256 bytes (cyclic)
ast2100::DecodeRC4_setup 0x0011b870 RC4 KSA (Key Scheduling Algorithm)
ast2100::RC4_crypt 0x0011b8f0 RC4 PRGA - XOR decrypt/encrypt
ast2100::WORD_hi_lo 0x0011b960 Byte order utility
ast2100::prepare_range_limit_table 0x0011b970 Build clamping table
ast2100::lookKbits 0x0011ba20 Peek N bits from bitstream
ast2100::skipKbits 0x0011ba40 Consume N bits from bitstream
ast2100::getKbits 0x0011bad0 Get N bits from bitstream
ast2100::init_QT 0x0011bb10 Init quantization table pointers
ast2100::set_quant_table 0x0011bb40 Set quantization table entries
ast2100::load_quant_table 0x0011bbb0 Load quantization table (Y)
ast2100::load_quant_tableCb 0x0011bd50 Load quantization table (Cb/Cr)
ast2100::load_advance_quant_table 0x0011bf70 Load advance mode QT
ast2100::load_advance_quant_tableCb 0x0011c0e0 Load advance mode QT (Cb/Cr)
ast2100::process_Huffman_data_unit 0x0011c430 Huffman decode one 8x8 block
ast2100::init_jpg_table 0x0011c6c0 Initialize JPEG Huffman/QT tables
ast2100::SetOptions 0x0011c770 Parse 4-byte header, init decoder state
ast2100::FreeQT 0x0011c990 Free quantization tables
ast2100::init_JPG_decoding 0x0011c940 Initialize JPEG decoding state
ast2100::updatereadbuf 0x0011ce30 Advance bitstream read position
ast2100::YUVToRGB 0x0011cec0 YCbCr -> BGRX32 color conversion
ast2100::Decompress 0x0011d150 Huffman+IDCT decompress one tile
ast2100::VQ_Decompress 0x0011d490 VQ decompress one tile
ast2100::MoveBlockIndex 0x0011d590 Advance tile position, record dirty rect
ast2100::VQ_Initialize 0x0011d670 Initialize 4-entry COLOR_CACHE
ast2100::decode 0x0011d6a0 Main decode entry - tile command loop
ast2100::get_decoderID 0x0011df60 Return decoder identifier
ast2100::get_ImageData 0x0011df00 Return decoded image data pointer
ast2100::PsudoStreamSwap32 0x0011df30 Byte-swap u32

RC4 key details:

  • ASCII: fedcba9876543210 (16 bytes at 0x00230500, null-terminated)
  • The key is expanded to 256 bytes by Keys_Expansion() which repeats the key cyclically: key[i] = key[i % strlen(key)] for i = 0..255
  • DecodeRC4_setup() is a standard RC4 KSA: initializes S-box [0..255], then permutes based on expanded key
  • RC4_crypt() is a standard RC4 PRGA: XOR each data byte with keystream

RC4 purpose: The RC4 encryption is used to decrypt the tile bitstream data when the encrypt_flag (this+0x101f6d) is set. It appears to be an optional obfuscation layer that some ASPEED AST2050 firmware versions enable. The first frame check (this+0x101e78 == 0) triggers key setup; subsequent frames reuse the RC4 state. The encryption flag can change between frames.

Hermon/Yarkon/Pilot3 Decoder Internals

HermonVideoDecoder functions:

Function Address Purpose
HermonVideoDecoder::HermonVideoDecoder (C1) 0x0010fc70 Constructor
HermonVideoDecoder::~HermonVideoDecoder (D1) 0x0010fd20 Destructor
HermonVideoDecoder::GetDecodedFrame 0x0010fd50 Return framebuffer ptr
HermonVideoDecoder::ConvertVierwerPixelFormat 0x0010fd60 16bpp/8bpp -> 32bpp
HermonVideoDecoder::SetRect 0x0010fe30 Copy 16x16 tile to framebuffer
HermonVideoDecoder::Decode 0x0010fee0 Main decode: full or incremental
HermonVideoDecoder::MixedCursor 0x00110120 Cursor compositing

YarkonVideoDecoder functions:

Function Address Purpose
YarkonVideoDecoder::YarkonVideoDecoder (C1) 0x001101c0 Constructor
YarkonVideoDecoder::~YarkonVideoDecoder (D1) 0x00110280 Destructor
YarkonVideoDecoder::GetDecodedFrame 0x001102b0 Return framebuffer ptr
YarkonVideoDecoder::AssignRectColour 0x001102c0 Fill rect with solid color
YarkonVideoDecoder::ConvertVierwerPixelFormat 0x00110380 RGB555 -> BGRX32
YarkonVideoDecoder::SetRect 0x00110400 Copy tile to framebuffer
YarkonVideoDecoder::HextileDecoder 0x001104a0 RFB Hextile tile decode
YarkonVideoDecoder::Decode 0x00110630 Main: Huffman/RLE + hextile
YarkonVideoDecoder::MixedCursor 0x00110850 Cursor compositing
Huffman_Compress 0x00121f40 Huffman compression (for capture)
Huffman_Uncompress 0x001221d0 Huffman decompression
RLE_Uncompress 0x0011a520 RLE decompression with escape chars

Pilot3VideoDecoder functions:

Function Address Purpose
Pilot3VideoDecoder::Pilot3VideoDecoder (C1) 0x00110900 Constructor
Pilot3VideoDecoder::~Pilot3VideoDecoder (D1) 0x001109e0 Destructor
Pilot3VideoDecoder::GetDecodedFrame 0x00110a20 Return framebuffer ptr
Pilot3VideoDecoder::ConvertVierwerPixelFormat 0x00110a30 Multi-format pixel convert
Pilot3VideoDecoder::SetRect 0x00110c20 Copy 32x32 tile to framebuffer
Pilot3VideoDecoder::Decode 0x00110d20 Main: multi-mode RLE + planar decode
Pilot3VideoDecoder::MixedCursor 0x00112490 Cursor compositing

Pilot3 object layout (0x458 bytes):

Offset  Size   Description
0x0C    2      Screen width (short)
0x0E    2      Screen height (short)
0x28    8      Working framebuffer pointer (converted pixels)
0x30    0x400  Color palette (256 entries x 4 bytes = 1024 bytes)
0x430   8      Decompression buffer pointer (8 MB)
0x438   8      Planar conversion buffer pointer (8 MB)
0x440   8      Input data pointer (compressed data)
0x448   8      Output framebuffer pointer (display)
0x450   4      Compressed data size
0x454   4      Total pixel data size (width * height * 4)

Screen Recording Subsystem

Global state variables:

Variable Address Purpose
startCapture 0x04923680 State machine: 0=idle, 1=started, 2=recording, -1=stopping
captureFile 0x04923628 FILE* for capture output
captureName 0x04923640 Filename string for capture
screenSource (BSS) Playback source: 0=none, 1=file loaded
sourceFile (BSS) FILE* for playback source
sourceName (BSS) Filename string for playback
playbackStatus (BSS) Playback state: 0=stopped, 1=playing, 2=stopped
isReplayRecord (BSS) Replay record flag

Recording flow:

  1. screenRecordingStartCapture(filename): Opens file for writing, sets startCapture=1, stores filename in captureName
  2. In RFBProtocol::ProtocolHandler(), when this->0x20 & 1 (capture flag): the FramebufferUpdate data is written to captureFile via fwrite()
  3. screenRecordingStopCapture(): Sets startCapture=-1 (stopping)

Playback flow:

  1. screenRecordingLoad(filename): Opens file for reading, sets screenSource=1, playbackStatus=1, isReplayRecord=1
  2. ProtocolHandler Mode 1: Copies sourceFile pointer to the TcpSocket's file mode (offset 0x44), causing reads to come from the file instead of the network socket
  3. screenRecordingStop(): Sets playbackStatus=2
  4. screenRecordingUnload(): Closes file, resets screenSource=0

Recording format: The recording is raw RFB protocol data -- the exact bytes received from the server (FramebufferUpdate messages). This is NOT an AVI or video format. The Java side (VideoRecorder class using ch.randelshofer.media.avi.AVIOutputStream) handles conversion to AVI format for user-facing recording. The native recording is for protocol-level replay.

Huffman_Compress (0x00121f40): Standard Huffman coding implementation:

  1. Build frequency table from input bytes
  2. Build Huffman tree by repeatedly combining lowest-frequency nodes
  3. Generate variable-length codes for each byte value
  4. Sort codes by frequency (bubble sort)
  5. Encode input using generated codes Returns compressed size minus start position.

Huffman_Uncompress (0x001221d0): Tree-based decompression:

  1. Reads the Huffman tree structure from the compressed data header
  2. For each output byte, traverses the tree bit-by-bit
  3. Leaf nodes contain the decoded byte values Both are used by the Yarkon decoder for its compressed data, not by the screen recording subsystem directly.

ProtocolHandler jump table:

Item Address Notes
Jump table base 0x00127790 61 entries, 4 bytes each (244 bytes)
Default handler 0x001183F5 Returns type byte, 54 entries point here
Type 0x00 target 0x0011849A Inline in ProtocolHandler
Type 0x04 target 0x001184DE Inline in ProtocolHandler
Type 0x16 target 0x00118488 Inline in ProtocolHandler
Type 0x35 target 0x0011844D Inline in ProtocolHandler
Type 0x37 target 0x00118457 Inline (falls through from 0x35)
Type 0x39 target 0x00118479 Inline in ProtocolHandler
Type 0x3C target 0x0011846A Inline in ProtocolHandler

Ghidra Analysis Notes

  • Both binaries are loaded and analyzed in the Ghidra project
  • libiKVM64.so binary name: /liblinux_x86_64__V1.0.5/libiKVM64.so-730126
  • libSharedLibrary64.so binary name: /liblinux_x86_64__V1.0.5/libSharedLibrary64.so-161223
  • C++ symbol demangling works well due to preserved .dynsym entries
  • Vtable analysis provides class hierarchy information
  • Many Java .class files are also loaded (useful for cross-referencing JNI)
  • Some duplicate function entries exist (DEFAULT vs IMPORTED sources) at different addresses -- the IMPORTED versions are the actual implementations
  • The ProtocolHandler jump table uses PC-relative offsets (each entry is a signed 32-bit offset from the table base at 0x00127790). Ghidra's decompiler could not recover the switch statement ("WARNING: Could not recover jumptable... Too many branches") so manual analysis of the raw bytes was required
  • RFBProtocol member offsets (0x50-0x68) are set by the constructors of RFBMouse, RFBKeyboard, RFBScreen, and RFBPrivilege respectively (each writes its this pointer back into the protocol object)

Keyboard Subsystem Analysis

Key finding: Keyboard events are NEVER encrypted. Despite having an RFBKMCryto* allocated at RFBKeyboard+0x20, the Sendkey() function never calls any encryption function. Only RFBMouse::SendMouse() uses AES.

RFBKeyboard Object Layout (total size ~0xB8):

Offset   Size   Type            Field              Notes
0x00     8      vtable*         vtable ptr
0x08     8      RMProtocol*     protocol           Network stream access
0x10     1      byte            config_received    Set to 1 by ProcKeyboardInfo
0x14     4      uint            keyboard_type      From ProcKeyboardInfo
0x18     4      uint            keyboard_mode      From ProcKeyboardInfo
0x20     8      RFBKMCryto*     crypto             AES instance (UNUSED by Sendkey)
0x28     48     std::map        vk_table_1         Table 1: printable chars (48 entries)
0x58     48     std::map        vk_table_2         Table 2: function/control keys (89 entries)
0x88     48     std::map        vk_table_3         Table 3: scan code map (32 entries)

processVK Key Translation Data Tables:

Table 1 data: 0x0022FD20 (48 x 8-byte records, 384 bytes) Table 2 data: 0x0022FEA0 (89 x 8-byte records, 712 bytes) Table 3 data: 0x00230180 (32 x 8-byte records, 256 bytes)

Each record is 8 bytes: [value:u32][key:u8, extended_flag:u8, pad:u16]

Table 2 uses the extended_flag byte: if non-zero, 0x100 is added to the key. Table 3 also uses the extended_flag byte in the same way.

Power Action Codes

Confirmed from JNI function decompilation:

JNI Function Address Action Code Description
setPowerOff 0x00121A70 0 Hard power off
setPowerOn 0x00121A50 1 Power on
setPowerReset 0x00121AB0 2 Hard reset
setSoftPowerOff 0x00121A90 3 ACPI soft off
getPowerStatus 0x00121AD0 (misnamed) Calls MouseGetPT (returns mouse config)

All call through vtable: desktop->screen->SetPowerOnOff(action) where SetPowerOnOff is RFBScreen vtable index 7 (offset 0x38) at 0x00118900.

RFBScreen Object Layout

The RFBScreen object is approximately 0x2080 bytes. Allocated by operator_new(0x2080).

Constructor (RFBScreen::RFBScreen at 0x001185d0):

  • Calls RMScreen::RMScreen(param_1) parent constructor
  • Sets frame_number (offset 0x50) = 0
  • Stores this pointer at param_1 + 0x60 (RFBProtocol.screen_handler)
  • Sets framebuffer ptr (0x38) = 0
  • Sets cursor_data_valid (0x55) = 0
  • Sets vtable to 0x0022F8D0
  • Allocates 0x600000 bytes (6,291,456 = ~6 MB) for compressed_data_buf (stored at 0x40)
  • Zeroes out cursor fields at 0x2058-0x206F (24 bytes)
Offset   Size   Type            Field                 Notes
0x00     8      vtable*         vtable ptr            0x0022F8D0
0x08     8      RMProtocol*     protocol              Set by RMScreen parent ctor
0x10     4      int             screen_width          Pixels (from ScreenDecode)
0x14     4      int             screen_height         Pixels (from ScreenDecode)
0x18     4      int             stored_width          For change detection (ScreenSetInfo)
0x1C     4      int             stored_height         For change detection (ScreenSetInfo)
0x20     4      uint            encoding_type         0x57-0x61 (from ScreenDecode)
0x28     8      uint64          update_blocks         Dirty rect tracking
0x38     8      uchar*          framebuffer           Decoded pixels (BGRX), set via ScreenSetFrameBuff
0x40     8      uchar*          compressed_data_buf   Incoming encoded data (0x600000 bytes allocated)
0x48     8      uchar*          cursor_buffer         Cursor Java ByteBuffer, set via ScreenSetFrameBuff
0x50     4      int             frame_number          Sequence counter (0 initially)
0x54     1      byte            resolution_changed    Notify flag (set when frame_number==1, first frame)
0x55     1      byte            cursor_data_valid     Cursor available (set when cursor_type==1 received)
0x56     varies  uchar[]        cursor_pixel_data     Inline cursor image (width*height*2 bytes)
0x2058   4      uint            cursor_x              Hardware cursor X position
0x205C   4      uint            cursor_y              Hardware cursor Y position
0x2060   4      int             cursor_width          Cursor image width
0x2064   4      int             cursor_height         Cursor image height
0x2070   4      uint            cursor_extra          Additional cursor info
0x2078   8      RMDecoder*      current_decoder       Active video decoder (from GetDecoder)

ast_jpeg Decoder State (0x588 bytes)

Offset   Size   Type       Field              Notes
0x08     8      uchar*     output_buffer      16bpp or 32bpp framebuffer
0x1C     4      int        cb_tile_w          Cb tile width
0x20     4      int        cb_tile_h          Cb tile height
0x24     4      int        y_tile_w           Y tile width
0x28     4      int        y_tile_h           Y tile height
0x30     8      void*      temp_buffer        Freed after decode
0x38     4      int        advance_mode       From header bit 31
0x3C     4      int        cbcr_quant         Chrominance quant level
0x40     4      int        advance_sign       Negated if advance_mode set
0x450    4      int        (zero init)
0x454    4      int        tile_x             Current tile X index
0x458    4      int        tile_y             Current tile Y index
0x45C    4      int        prev_tile_x
0x460    4      int        prev_tile_y
0x468    8      uint64     decoded_pixels     Running count
0x470    8      uint64     (bitstream state)
0x478    8      uint64     codebuf            Current code buffer
0x480    8      uint64     newbuf             Next buffer
0x48C    4      int        newbits            Bits remaining
0x49C    4      uint       orig_width
0x4A0    4      uint       orig_height
0x4BC    4      uint       cbcr_tile_size     From header
0x518    8      ulong*     buffer_ptr         Data buffer pointer
0x520    8      uchar*     decoded_pixels_buf 32bpp decoded tile data
0x528    8      uint64     buffer_index       Current position in buffer
0x530    8      long       aligned_width      Width aligned to tile boundary
0x538    8      long       aligned_height     Height aligned to tile boundary
0x540    8      long       padded_height
0x548    8      long       padded_width
0x550    1      byte       (config)
0x551    1      byte       Y_DC_selector      = 1
0x552    1      byte       Y_AC_selector      = 1
0x553    1      byte       Y_DC_idx           = 0
0x554    1      byte       Cb_DC_idx          = 1
0x555    1      byte       Cr_DC_idx          = 1
0x556    1      byte       Y_AC_idx           = 0
0x557    1      byte       Cb_AC_idx          = 1
0x558    1      byte       Cr_AC_idx          = 1
0x55A    2      short      Y_DC_prev          DC coefficient accumulator (Y)
0x55C    2      short      Cb_DC_prev         DC coefficient accumulator (Cb)
0x55E    2      short      Cr_DC_prev         DC coefficient accumulator (Cr)
0x564    4      int        tile_pixel_size    16 or other
0x568    4      int        R multiplier       Pixel format
0x56C    4      int        G multiplier
0x570    4      int        B multiplier
0x574    4      int        R shift
0x578    4      int        G shift
0x57C    4      int        B shift

Additional Discovered Functions

Keyboard subsystem:

Function Address Purpose
RFBKeyboard::Sendkey(int, int) 0x0011A670 Send key event (always cleartext)
RFBKeyboard::processVK(KeyInfo_t) 0x0011A920 VK -> X11 keysym translation
RFBKeyboard::KeyboardAction(KeyInfo_t) 0x0011A5D0 Entry: calls processVK via vtable
RFBKeyboard::KeyboardSync() 0x0011A5F0 Stub (returns 1)
RFBKeyboard::KeyboardReset() 0x0011A600 Reset keyboard state
RFBKeyboard::KeyboardUpdateInfo() 0x0011A610 Send type 0x35 request
RFBKeyboard::KeyboardGetType() 0x0011A650 Get keyboard type
RFBKeyboard::KeyboardSetType(KeyTypeInfo_t) 0x0011A660 Set keyboard type
RFBKeyboard::RFBKeyboard(RMProtocol*) 0x0011ABF0 / 0x0011AF50 Constructor (populates maps)
Java_..._keyboardAction 0x00121000 JNI entry: lock key + processVK

RFBScreen internals:

Function Address Purpose
RFBScreen::ScreenDecode() 0x00119090 Receive + decode FramebufferUpdate
RFBScreen::ScreenUpdate(ScreenReqInfo_t) 0x00118850 Send FramebufferUpdateRequest
RFBScreen::ScreenGetFreame() 0x00118D40 Return framebuffer pointer
RFBScreen::ScreenGetRect(ScreenReqInfo_t, uchar*) 0x00118A20 Copy rect from framebuffer
RFBScreen::ScreenSetFrameBuff(uchar*, uchar*) 0x00118D50 Set frame/cursor buffer ptrs
RFBScreen::ScreenSetInfo(ScreenInfo_t) 0x00118B80 Send resolution (type 0x32)
RFBScreen::ScreenCursorPosProc() 0x00118FB0 Handle cursor position (type 0x04)
RFBScreen::GeFrontGround() 0x00118DA0 Get cursor composite buffer
RFBScreen::GetCursorPos() 0x00118D60 Send type 0x19 (request cursor)
RFBScreen::ScreenGetInfo() 0x00118C10 Get current screen info struct

Video decoder chain:

Function Address Purpose
RMDecoder::GetDecoder(Decodeinfo, uchar*, uchar*) 0x00112560 Decoder factory (singleton)
ASTVideoDecoder::Decode(UpdateBlocks_t&) 0x0010F8B0 AST decode entry (dispatches to ast_jpeg or ast2100)
ASTVideoDecoder::GetDecodedFrame() 0x0010F8A0 Return decoded frame pointer
ast_jpeg::decode(ASTDecodeInfo&, UpdateBlocks_t&) 0x0011EA70 AST2100+ JPEG decode
ast_jpeg::SetOptions(ASTDecodeInfo&) 0x0011E280 Configure from frame header
ast_jpeg::YUVToRGB(...) 0x0011E4E0 YCbCr -> BGRX conversion
ast_jpeg::MoveBlockIndex() 0x0011E700 Advance tile X/Y index
ast_jpeg::ScreenResolution() 0x0011E770 Write decoded tile to output buffer
ast_jpeg::updatereadbuf(...) 0x0010E048 Advance bitstream read pointer
ast_jpeg::setScreenInfo(...) 0x0011F900 Configure screen parameters
ast_jpeg::get_buffer() 0x0011F8C0 Get compressed data buffer
ast_jpeg::get_ImageData() 0x0011F8D0 Get decoded image data
ast_jpeg::get_decoderID() 0x0011F990 Get decoder ID
ast_jpeg::PsudoStreamSwap16(ushort) 0x0011F940 16-bit byte swap
ast_jpeg::PsudoStreamSwap32(uint) 0x0011F960 32-bit byte swap
ast_jpeg::flag 0x00230FB0 Static: init_jpg_table needed flag

AES encryption:

Function Address Purpose
RFBKMCryto::EnCryto(char*, char*, int) 0x00117050 Encrypt wrapper
RFBKMCryto::DeCryto(...) 0x001149C0 Decrypt wrapper
RFBKMCryto::RFB_AES128_EventCryto(uchar*, uchar*) 0x00116E30 AES-128-CBC event encrypt
RFBKMCryto::SW_AES_CBC(mode, keysize, input, blocks, key, output, iv) 0x0010E2E8 Software AES-CBC
RFBKMCryto::aes_encrypt(aes_context*, uchar*, uchar*) 0x00114F20 AES block encrypt
RFBKMCryto::aes_set_key(aes_context*, uchar*, int) 0x0010E278 Set key (128/192/256)
RFBKMCryto::aes_gen_tables() 0x001149D0 Generate S-boxes
RFBKMCryto::SetCipherKey(...) 0x0011E250 Configure cipher key
RFBKMCryto::RSb 0x001225E0 Reverse S-box table

Power control:

Function Address Action Code Purpose
setPowerOn 0x00121A50 1 Power on
setPowerOff 0x00121A70 0 Hard power off
setSoftPowerOff 0x00121A90 3 ACPI soft power off
setPowerReset 0x00121AB0 2 Hard reset
getPowerStatus 0x00121AD0 -- Misnamed: calls MouseGetPT() (returns mouse config)
RFBScreen::SetPowerOnOff(uchar) 0x00118900 -- Send type 0x1A + action byte