tokenize
Convert Pylpex code into tokens for analysis by breaking down source code into identifiable components with types, values, and positions.
Instructions
Convert Pylpex code into tokens for analysis.
Args: code: Valid Pylpex source code to tokenize
Returns: List of tokens with their types, values, and positions
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| code | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| count | No | ||
| error | No | ||
| tokens | Yes | ||
| success | No | ||
| code_analyzed | No | ||
| code_attempted | No |