# Scrcpy Control Protocol Specification (v2.x/v3.x)
## Protocol Overview
The scrcpy control protocol is a binary protocol used for bidirectional communication between the client and server (Android device). The protocol operates on a dedicated socket connection (separate from the video stream).
**Version**: 2.x/3.x (current master branch)
**Research Date**: 2025-12-22
## Connection Architecture
### Socket Connection Sequence
1. **Initial Connection**: Two sockets are established after ADB connection:
- **Video Socket**: Receives H.264/H.265 video stream
- **Control Socket**: Bidirectional control messages (THIS PROTOCOL)
2. **Socket Type**: TCP socket via ADB tunnel (localhost)
3. **Control Channel**: Created with LocalSocket on Android side
- `ControlChannel.java` wraps the socket
- Uses `DataInputStream` and `DataOutputStream` for reading/writing
- Uses `BufferedInputStream` and `BufferedOutputStream` for efficiency
### Message Flow
- **Client → Server**: Control messages (input events)
- **Server → Client**: Device messages (clipboard, acknowledgments)
---
## Control Message Types (Client → Server)
### Message Type Constants
```c
enum sc_control_msg_type {
SC_CONTROL_MSG_TYPE_INJECT_KEYCODE = 0,
SC_CONTROL_MSG_TYPE_INJECT_TEXT = 1,
SC_CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT = 2,
SC_CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT = 3,
SC_CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON = 4,
SC_CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL = 5,
SC_CONTROL_MSG_TYPE_EXPAND_SETTINGS_PANEL = 6,
SC_CONTROL_MSG_TYPE_COLLAPSE_PANELS = 7,
SC_CONTROL_MSG_TYPE_GET_CLIPBOARD = 8,
SC_CONTROL_MSG_TYPE_SET_CLIPBOARD = 9,
SC_CONTROL_MSG_TYPE_SET_DISPLAY_POWER = 10,
SC_CONTROL_MSG_TYPE_ROTATE_DEVICE = 11,
SC_CONTROL_MSG_TYPE_UHID_CREATE = 12,
SC_CONTROL_MSG_TYPE_UHID_INPUT = 13,
SC_CONTROL_MSG_TYPE_UHID_DESTROY = 14,
SC_CONTROL_MSG_TYPE_OPEN_HARD_KEYBOARD_SETTINGS = 15,
SC_CONTROL_MSG_TYPE_START_APP = 16,
SC_CONTROL_MSG_TYPE_RESET_VIDEO = 17,
};
```
---
## Binary Message Formats
All multi-byte values use **Big-Endian (Network) byte order**.
### 1. INJECT_KEYCODE (Type 0)
**Total Size**: 14 bytes
```
Byte 0: Type (1 byte) = 0
Byte 1: Action (1 byte)
Bytes 2-5: Keycode (uint32, big-endian)
Bytes 6-9: Repeat count (uint32, big-endian)
Bytes 10-13: Meta state (uint32, big-endian)
```
**Action Values** (Android KeyEvent):
- 0 = ACTION_DOWN
- 1 = ACTION_UP
- 2 = ACTION_MULTIPLE
**Keycode Values**: Android KeyEvent constants (e.g., KEYCODE_HOME = 3, KEYCODE_BACK = 4)
**Meta State**: Combination of:
- 0x00000001 = META_SHIFT_ON
- 0x00000002 = META_ALT_ON
- 0x00000004 = META_SYM_ON
- 0x00000008 = META_FUNCTION_ON
- 0x00000010 = META_ALT_LEFT_ON
- 0x00000020 = META_ALT_RIGHT_ON
- 0x00000040 = META_SHIFT_LEFT_ON
- 0x00000080 = META_SHIFT_RIGHT_ON
- 0x00000100 = META_CTRL_ON
- 0x00000200 = META_CTRL_LEFT_ON
- 0x00000400 = META_CTRL_RIGHT_ON
- 0x00000800 = META_META_ON
- 0x00001000 = META_META_LEFT_ON
- 0x00002000 = META_META_RIGHT_ON
- 0x00004000 = META_CAPS_LOCK_ON
- 0x00008000 = META_NUM_LOCK_ON
- 0x00010000 = META_SCROLL_LOCK_ON
### 2. INJECT_TEXT (Type 1)
**Variable Size**: 1 + 4 + len (max 304 bytes)
```
Byte 0: Type (1 byte) = 1
Bytes 1-4: String length (uint32, big-endian)
Bytes 5+: UTF-8 text (NOT null-terminated)
```
**Constraints**:
- Maximum text length: 300 characters
- Encoding: UTF-8
- No null terminator
### 3. INJECT_TOUCH_EVENT (Type 2)
**Total Size**: 32 bytes
```
Byte 0: Type (1 byte) = 2
Byte 1: Action (1 byte)
Bytes 2-9: Pointer ID (uint64, big-endian)
Bytes 10-13: X coordinate (int32, big-endian)
Bytes 14-17: Y coordinate (int32, big-endian)
Bytes 18-19: Screen width (uint16, big-endian)
Bytes 20-21: Screen height (uint16, big-endian)
Bytes 22-23: Pressure (u16fp, see below)
Bytes 24-27: Action button (uint32, big-endian)
Bytes 28-31: Buttons (uint32, big-endian)
```
#### Touch Event Actions
```c
ACTION_DOWN = 0
ACTION_UP = 1
ACTION_MOVE = 2
ACTION_CANCEL = 3
ACTION_OUTSIDE = 4
ACTION_POINTER_DOWN = 5
ACTION_POINTER_UP = 6
ACTION_HOVER_MOVE = 7
ACTION_HOVER_ENTER = 9
ACTION_HOVER_EXIT = 10
ACTION_BUTTON_PRESS = 11
ACTION_BUTTON_RELEASE = 12
```
#### Pointer ID Special Values
```c
SC_POINTER_ID_MOUSE = UINT64_C(-1) // 0xFFFFFFFFFFFFFFFF
SC_POINTER_ID_GENERIC_FINGER = UINT64_C(-2) // 0xFFFFFFFFFFFFFFFE
SC_POINTER_ID_VIRTUAL_FINGER = UINT64_C(-3) // 0xFFFFFFFFFFFFFFFD
```
**Multiple Touch**: For multi-touch, each additional pointer needs an ACTION_POINTER_DOWN with the pointer index encoded:
```
action = ACTION_POINTER_DOWN | (pointer_index << ACTION_POINTER_INDEX_SHIFT)
```
#### Pressure Format (u16fp)
Pressure is encoded as a 16-bit fixed-point number:
- Range: [0.0, 1.0]
- Formula: `u16fp_value = (uint16_t)(pressure * 65535.0f)`
- Decoding: `pressure = u16fp_value / 65535.0f`
#### Button Flags
```c
BUTTON_PRIMARY = 1 // 0x00000001
BUTTON_SECONDARY = 2 // 0x00000002
BUTTON_TERTIARY = 4 // 0x00000004
BUTTON_BACK = 8 // 0x00000008
BUTTON_FORWARD = 16 // 0x00000010
BUTTON_STYLUS = 32 // 0x00000020
BUTTON_STYLUS2 = 64 // 0x00000040
BUTTON_TOUCH = 256 // 0x00000100
```
### 4. INJECT_SCROLL_EVENT (Type 3)
**Total Size**: 21 bytes
```
Byte 0: Type (1 byte) = 3
Bytes 1-4: X coordinate (int32, big-endian)
Bytes 5-8: Y coordinate (int32, big-endian)
Bytes 9-10: Screen width (uint16, big-endian)
Bytes 11-12: Screen height (uint16, big-endian)
Bytes 13-14: Horizontal scroll (i16fp, see below)
Bytes 15-16: Vertical scroll (i16fp, see below)
Bytes 17-20: Buttons (uint32, big-endian)
```
#### Scroll Value Format (i16fp)
Scroll values are encoded as 16-bit signed fixed-point numbers:
- Range: [-16, 16] (not [-1, 1])
- Formula: `i16fp_value = (int16_t)(scroll_value * 2048.0f)` [when normalizing to [-1, 1]]
- The actual range is [-16, 16], so values are divided by 16 before normalization
- Decoding: `scroll_value = (i16fp_decoded_as_16bit / 32768.0f) * 16`
**Implementation Note**: In control_msg.c:
```c
float hscroll_norm = msg->inject_scroll_event.hscroll / 16;
hscroll_norm = CLAMP(hscroll_norm, -1, 1);
int16_t hscroll = sc_float_to_i16fp(hscroll_norm);
```
### 5. BACK_OR_SCREEN_ON (Type 4)
**Total Size**: 2 bytes
```
Byte 0: Type (1 byte) = 4
Byte 1: Action (1 byte, 0=DOWN or 1=UP)
```
### 6. EXPAND_NOTIFICATION_PANEL (Type 5)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 5
```
No additional data.
### 7. EXPAND_SETTINGS_PANEL (Type 6)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 6
```
No additional data.
### 8. COLLAPSE_PANELS (Type 7)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 7
```
No additional data.
### 9. GET_CLIPBOARD (Type 8)
**Total Size**: 2 bytes
```
Byte 0: Type (1 byte) = 8
Byte 1: Copy key (1 byte)
```
**Copy Key Values**:
- 0 = COPY_KEY_NONE (don't send any key)
- 1 = COPY_KEY_COPY (send Ctrl+C before reading)
- 2 = COPY_KEY_CUT (send Ctrl+X before reading)
### 10. SET_CLIPBOARD (Type 9)
**Variable Size**: 1 + 8 + 1 + 4 + len
```
Byte 0: Type (1 byte) = 9
Bytes 1-8: Sequence number (uint64, big-endian)
Byte 9: Paste flag (0 or 1)
Bytes 10-13: String length (uint32, big-endian)
Bytes 14+: UTF-8 text (NOT null-terminated)
```
**Constraints**:
- Maximum clipboard text: 262,136 bytes (256KB - 14 bytes overhead)
- Encoding: UTF-8
- Sequence number for acknowledgment (0 = invalid/no ack)
### 11. SET_DISPLAY_POWER (Type 10)
**Total Size**: 2 bytes
```
Byte 0: Type (1 byte) = 10
Byte 1: Power state (0=off, 1=on)
```
### 12. ROTATE_DEVICE (Type 11)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 11
```
No additional data.
### 13. UHID_CREATE (Type 12)
**Variable Size**: 1 + 2 + 2 + 2 + 1 + name_len + 2 + desc_len
```
Byte 0: Type (1 byte) = 12
Bytes 1-2: Device ID (uint16, big-endian)
Bytes 3-4: Vendor ID (uint16, big-endian)
Bytes 5-6: Product ID (uint16, big-endian)
Byte 7: Name length (uint8)
Bytes 8+: Device name (UTF-8, max 127 bytes)
Bytes n: Report descriptor size (uint16, big-endian)
Bytes n+2+: Report descriptor binary data
```
### 14. UHID_INPUT (Type 13)
**Variable Size**: 1 + 2 + 2 + len (max ~8KB)
```
Byte 0: Type (1 byte) = 13
Bytes 1-2: Device ID (uint16, big-endian)
Bytes 3-4: Data size (uint16, big-endian)
Bytes 5+: HID report data (binary)
```
### 15. UHID_DESTROY (Type 14)
**Total Size**: 3 bytes
```
Byte 0: Type (1 byte) = 14
Bytes 1-2: Device ID (uint16, big-endian)
```
### 16. OPEN_HARD_KEYBOARD_SETTINGS (Type 15)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 15
```
No additional data.
### 17. START_APP (Type 16)
**Variable Size**: 1 + 1 + len (max 256 bytes)
```
Byte 0: Type (1 byte) = 16
Byte 1: Name length (uint8)
Bytes 2+: Package name (UTF-8, max 255 bytes)
```
**Special Prefixes**:
- `?name`: Search for app by display name
- `+name`: Force stop before starting
- `+?name`: Force stop, then search by display name
### 18. RESET_VIDEO (Type 17)
**Total Size**: 1 byte
```
Byte 0: Type (1 byte) = 17
```
No additional data.
---
## Position Format (Used in Touch & Scroll Events)
Position structure appears in touch and scroll events:
```c
struct sc_position {
struct {
int32_t x;
int32_t y;
} point;
struct {
uint16_t width;
uint16_t height;
} screen_size;
};
```
**Bytes Layout** (12 bytes total):
```
Bytes 0-3: X coordinate (int32, big-endian)
Bytes 4-7: Y coordinate (int32, big-endian)
Bytes 8-9: Screen width (uint16, big-endian)
Bytes 10-11: Screen height (uint16, big-endian)
```
**Coordinate Notes**:
- Coordinates are absolute screen positions (not deltas)
- Screen size must match the current device resolution
- Coordinates are mapped to actual device display via PositionMapper
---
## Maximum Message Sizes
```c
#define SC_CONTROL_MSG_MAX_SIZE (1 << 18) // 256KB
#define SC_CONTROL_MSG_INJECT_TEXT_MAX_LENGTH 300
#define SC_CONTROL_MSG_CLIPBOARD_TEXT_MAX_LENGTH (SC_CONTROL_MSG_MAX_SIZE - 14)
// = 262,136 bytes (256KB - overhead)
#define SC_HID_MAX_SIZE 4096
```
---
## Connection Handshake
### Initial Setup (Server-Side - Java)
```java
// From Controller.java
ControlChannel controlChannel;
ControlMessageReader reader;
ControlMessageWriter writer;
// Reading from client
ControlMessage msg = controlChannel.recv(); // Blocking read
// Writing to client
DeviceMessage msg = DeviceMessage.createClipboard(text);
controlChannel.send(msg);
```
### Socket Initialization
1. After ADB tunnel connects, client creates socket connection
2. Server opens `LocalSocket` for control input
3. Both client and server enter message loop:
- Client sends control messages (user input)
- Server sends device messages (clipboard, etc.)
### No Handshake/Greeting Required
- Protocol starts immediately with first message
- Type byte determines message structure
- Unknown types cause "Unknown event type" error on server
---
## Data Type Encoding Reference
### Fixed-Point Number Encoding
**u16fp (unsigned 16-bit fixed-point)**:
- Used for pressure values [0.0, 1.0]
- Encoding: `value = (uint16_t)(float_value * 65535.0f)`
- Decoding: `float_value = uint16_value / 65535.0f`
- Range: 0x0000 (0.0) to 0xFFFF (1.0)
**i16fp (signed 16-bit fixed-point)**:
- Used for scroll values [-1.0, 1.0] (then multiplied by 16 for [-16, 16])
- Encoding: `value = (int16_t)(float_value * 32767.0f)`
- Decoding: `float_value = int16_value / 32767.0f`
- Range: 0x8000 (-1.0) to 0x7FFF (1.0)
### String Encoding
**4-byte length prefix** (for text and clipboard):
```c
uint32_t length; // 4 bytes, big-endian
uint8_t data[length]; // UTF-8 bytes, NO null terminator
```
**1-byte length prefix** (for name in UHID and START_APP):
```c
uint8_t length; // 1 byte
uint8_t data[length]; // UTF-8 bytes, NO null terminator
```
---
## Important Protocol Notes
### Pointer State Management
- Server maintains `PointersState` tracking all active pointers
- Maximum 10 simultaneous pointers (PointersState.MAX_POINTERS)
- Each pointer has local ID (0-9) for MotionEvent.PointerProperties
- Pointer IDs from client are preserved but mapped to local IDs
### Touch Event Sequencing
For multi-touch:
1. First pointer sends ACTION_DOWN
2. Additional pointers send ACTION_POINTER_DOWN (with index encoded in action)
3. All pointers send ACTION_MOVE when position changes
4. All pointers send ACTION_UP/ACTION_POINTER_UP when lifted
5. Last pointer lifted triggers final ACTION_UP
### Pressure and Size
- **Pressure**: 0.0 = no pressure, 1.0 = maximum pressure
- **Size**: Set to 0 for finger touches (MotionEvent.PointerCoords.size)
- **Orientation**: Set to 0 (MotionEvent.PointerCoords.orientation)
### Display ID Handling
- Touches/scrolls sent to virtual display ID (from video capture)
- Key events sent to main display or virtual display depending on mode
- Position mapping handles different resolution/orientation via PositionMapper
### Timestamp Note
- Server generates timestamps internally (SystemClock.uptimeMillis())
- Client does NOT send timestamps in control messages
- Server tracks lastTouchDown for multi-pointer MotionEvents
---
## Error Handling
### On Server Receiving Unknown Type
```java
case ControlMessage.TYPE_UNKNOWN:
throw new ControlProtocolException("Unknown event type: " + type);
```
### On Protocol Errors
- IOException during recv/send terminates control stream
- Malformed messages cause exceptions
- Invalid pointer counts logged as warnings
---
## Implementation Requirements for Custom Control Socket
### Serialization (Client → Server)
1. Start with type byte (0-17)
2. Serialize remaining fields in big-endian
3. Fixed-point values encoded as described
4. String length as 4-byte or 1-byte prefix, UTF-8 data follows
5. Write to socket OutputStream
### Deserialization (Server reading from client)
1. Read type byte
2. Switch on type to determine remaining bytes to read
3. Use DataInputStream for all multi-byte reads (auto big-endian)
4. Validate pressure/scroll ranges
5. Map coordinates using PositionMapper
### Socket Buffer Requirements
- Read buffer: ≥256KB (for clipboard messages)
- Write buffer: ≥256KB (for large clipboard reads)
- Use buffered streams for efficiency (BufferedInputStream/BufferedOutputStream)
---
## Example: Sending a Tap
```
Message: TAP at (500, 1000) on 1080x1920 device
Type (0-0): 0x02 // INJECT_TOUCH_EVENT
Action (1-1): 0x00 // ACTION_DOWN
Pointer ID (2-9): 0xFFFFFFFFFFFFFFFF // MOUSE
X (10-13): 0x000001F4 // 500 (big-endian)
Y (14-17): 0x000003E8 // 1000 (big-endian)
Width (18-19): 0x0438 // 1080 (big-endian)
Height (20-21): 0x0780 // 1920 (big-endian)
Pressure (22-23): 0xFFFF // 1.0 (max)
Action Button (24-27): 0x00000001 // PRIMARY
Buttons (28-31): 0x00000001 // PRIMARY
[Send 32 bytes above]
Type (0-0): 0x02 // INJECT_TOUCH_EVENT
Action (1-1): 0x01 // ACTION_UP
Pointer ID (2-9): 0xFFFFFFFFFFFFFFFF // MOUSE
X (10-13): 0x000001F4 // 500
Y (14-17): 0x000003E8 // 1000
Width (18-19): 0x0438 // 1080
Height (20-21): 0x0780 // 1920
Pressure (22-23): 0x0000 // 0.0
Action Button (24-27): 0x00000000 // NONE
Buttons (28-31): 0x00000000 // NONE
[Send 32 bytes above]
```
---
## References
**Source Files**:
- Server: `/server/src/main/java/com/genymobile/scrcpy/control/`
- `ControlMessage.java` - Message definitions
- `ControlMessageReader.java` - Parsing
- `Controller.java` - Event processing
- `Pointer.java` - Pointer state
- `ControlChannel.java` - Socket wrapper
- Client: `/app/src/control_msg.h` and `/app/src/control_msg.c`
- Message serialization
- Type definitions
- Binary format details
**Protocol Features** (v2.x/v3.x):
- Multi-touch support (10 pointers max)
- UHID keyboard/mouse virtual devices
- Clipboard bidirectional sync
- Display power control
- Device rotation
- App launching
- Video reset signaling
