tokenize
Convert Pylpex code into tokens for analysis by breaking down source code into identifiable components with types, values, and positions.
Instructions
Convert Pylpex code into tokens for analysis.
Args: code: Valid Pylpex source code to tokenize
Returns: List of tokens with their types, values, and positions
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| code | Yes |