Smooth Operator Agent Tools

""" Installation and Usage Guide for the Smooth Operator Agent Tools Python Library """ # Installation ## Using pip The Smooth Operator Agent Tools Python library can be installed using pip: ```bash pip install smooth-operator-agent-tools ``` This will automatically install the library and all its dependencies, including the server executable. ## From Source To install from source: 1. Clone the repository: ```bash git clone https://github.com/fstandhartinger/smooth-operator-client-python.git cd smooth-operator-client-python ``` 2. Install the package: ```bash pip install -e . ``` # Basic Usage ## Initializing the Client ```python from smooth_operator_agent_tools import SmoothOperatorClient # Initialize the client client = SmoothOperatorClient(api_key="YOUR_API_KEY") # Get API key for free at https://screengrasp.com/api.html # Start the server client.start_server() # Stop the server when done client.stop_server() ``` You can also use the client as a context manager: ```python from smooth_operator_agent_tools import SmoothOperatorClient with SmoothOperatorClient() as client: client.start_server() # Use the client here # Server will be automatically stopped when exiting the context ``` ## Taking Screenshots ```python # Take a screenshot - returns image in a form that can easily be passed to LLMs screenshot = client.screenshot.take() # Access the screenshot data image_bytes = screenshot.image_bytes image_base64 = screenshot.image_base64 ``` ## Mouse Operations ```python # Click at coordinates client.mouse.click(500, 300) # Right-click at coordinates client.mouse.right_click(500, 300) # Double-click at coordinates client.mouse.double_click(500, 300) # Drag from one position to another client.mouse.drag(100, 100, 200, 200) # Scroll at coordinates client.mouse.scroll(500, 300, 5) # Scroll down 5 clicks client.mouse.scroll(500, 300, -5) # Scroll up 5 clicks ``` ## AI-Powered UI Interaction ```python # Find and click a UI element by description client.mouse.click_by_description("the Submit button") # Find and right-click a UI element by description client.mouse.right_click_by_description("the Context menu icon") # Find and double-click a UI element by description client.mouse.double_click_by_description("the File icon") # Drag from one element to another by description client.mouse.drag_by_description("the invoice pdf file", "the 'invoices' folder") ``` ## Keyboard Operations ```python # Type text client.keyboard.type("Hello, world!") # Press a key combination client.keyboard.press("Ctrl+C") client.keyboard.press("Alt+F4") # Type text in a UI element client.keyboard.type_at_element("the Username field", "user123") ``` ## Chrome Browser Control ```python # Open Chrome browser client.chrome.open_chrome("https://www.example.com") # Navigate to a URL client.chrome.navigate("https://www.google.com") # Get information about the current tab # Can be used to find likely interactable elements in the page # Marks all html elements with robust CSS selectors for use # in functions like click_element() or simulate_input() # Response can also be passed to LLM to pick the right selector tab_details = client.chrome.explain_current_tab() # Click an element using CSS selector client.chrome.click_element("#search-button") # Input text into a form field client.chrome.simulate_input("#username", "user123") # Execute JavaScript result = client.chrome.execute_script("return document.title") # Generate and execute JavaScript based on a description result = client.chrome.generate_and_execute_script("Extract all links from the page") ``` ## System Operations ```python # Get system overview # Contains list of windows, available apps on the system, # detailed infos about the currently focused ui element and window. # Can be used as a source of ui element ids for use in automation functions # like invoke() (=click) or set_value(). # Can be used as a source of window ids for get_window_details(window_id). # Consider sending the json serialized form of this result to a LLM, together # with a task description, the form is chosen to be LLM friendly, the LLM # sould be able to find the relevand ui element ids and windows ids like that. overview = client.system.get_overview() # Open an application client.system.open_application("notepad") # Get window details - contains the ui automation tree of elements. # Consider using the response in a LLM prompt. window_id = overview.windows[0].id window_details = client.system.get_window_details(window_id) ``` ## Windows Automation ```python # Click a UI element by description # element ids can be acquired from get_overview() and get_window_details() client.automation.invoke(element_id) # Type text in a UI element # element ids can be acquired from get_overview() and get_window_details() client.automation.set_value(element_id, "john doe") # Bring a window to the front client.automation.bring_to_front(window_id) ``` ## Code Execution ```python # Execute C# code result = client.code.execute_csharp("return 2 + 2;") # Generate and execute C# code based on a description - example 1 result = client.code.generate_and_execute_csharp("Calculate the factorial of 5") # Generate and execute C# code based on a description - example 2 result = client.code.generate_and_execute_csharp("Return content of the biggest file in folder c:\\temp") # Generate and execute C# code based on a description - example 3 result = client.code.generate_and_execute_csharp("Connect to Outlook via Interop and return text and date of the latest email from pricelist@vendor.com") ``` # Advanced Usage ## Using Different AI Mechanisms For AI-vision powered operations (provided by Screengrasp.com), you can specify different AI mechanisms: ```python from smooth_operator_agent_tools import MechanismType # Use a different AI mechanism client.mouse.click_by_description("the Submit button", mechanism=MechanismType.OPENAI_COMPUTER_USE) ``` ## Converting Responses to JSON - use LLMs to analyze Most response objects have a `to_json_string()` method that converts the response to a JSON string: ```python # Get a response screenshot = client.screenshot.take() # Convert to JSON string json_str = screenshot.to_json_string() # Use the JSON string (e.g., pass it to a language model) print(json_str) ``` It is a recommended pattern to use these JSON strings with LLMs to analyze the content. For example you can prompt GPT-4o to extract the CSS selector of "the UI element that can be clicked to submit the form" by providing a textual instruction and the JSON string in a prompt. Use GPT-4o's JSON mode (for some LLMs also called structured output) to ensure it answers in a form you can easily parse. # Platform Support The Smooth Operator Agent Tools Python library is designed to work on Windows platforms, as the server executable is a Windows application. Support for other platforms may be added in the future.