> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spongecake.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Agent Integration

> Integrate OpenAI with your desktop client for higher-level agent logic.

### **`action()`**

> Check out the [guide for using this function](#🚀-guide%3A-using-the-action-command) for more details

### Purpose

The `action` function lets you control the desktop environment via an agent, managing commands, user inputs, security checks, and tool calls in a streamlined way.

### Arguments

* **`input_text`** *(str, optional)*:\
  New commands or responses to agent prompts.

* **`acknowledged_safety_checks`** *(bool, optional)*:\
  Set `True` after the user confirms pending security checks.

* **`ignore_safety_and_input`** *(bool, optional)*:\
  Automatically approves security checks and inputs. **Use cautiously.**

* **Handlers** *(callables, optional)*:\
  Customize how different statuses are handled:
  * **`complete_handler(data)`**: Final results.
  * **`needs_input_handler(messages)`**: Collects user input.
  * **`needs_safety_check_handler(safety_checks, pending_call)`**: Approves security checks.
  * **`error_handler(error_message)`**: Manages errors.

* **`tools`** *(list, optional)*:\
  List of available tools for the agent to use when executing actions.

* **`function_map`** *(dict, optional)*:\
  Dictionary mapping tool call identifiers to actual callable functions.

### How it works

The `action` function returns one of four statuses:

### Status Handling

* **COMPLETE**:\
  Task finished successfully. Handle final output.

* **ERROR**:\
  Review the returned error message and handle accordingly.

* **NEEDS\_INPUT**:\
  Provide additional user input and call `action()` again with this input.

* **NEEDS\_SECURITY\_CHECK**:\
  Review security warnings and confirm with `acknowledged_safety_checks=True`.

Example workflow:

```python theme={null}
status, data = agent.action(input_text="Open Chrome")

if status == AgentStatus.COMPLETE:
    print("Done:", data)
elif status == AgentStatus.ERROR:
    print("Error:", data)
elif status == AgentStatus.NEEDS_INPUT:
    user_reply = input(f"Input needed: {data}")
    agent.action(input_text=user_reply)
elif status == AgentStatus.NEEDS_SECURITY_CHECK:
    confirm = input(f"Security checks: {data['safety_checks']} Proceed? (y/N): ")
    if confirm.lower() == "y":
        agent.action(acknowledged_safety_checks=True)
    else:
        print("Action cancelled.")
```

### Auto Mode

Set `ignore_safety_and_input=True` for automatic handling of inputs and security checks. Use carefully as this bypasses user prompts and approvals.

### Using Handlers

Provide handler functions to automate status management, simplifying your code:

```python theme={null}
agent.action(
    input_text="Open Chrome",
    complete_handler=lambda data: print("Done:", data),
    error_handler=lambda error: print("Error:", error),
    needs_input_handler=lambda msgs: input(f"Agent asks: {msgs}"),
    needs_safety_check_handler=lambda checks, call: input(f"Approve {checks}? (y/N): ").lower() == "y"
)
```

### Using Tools and Function Map

When providing tools, ensure each tool has a corresponding callable in the `function_map`. The agent will invoke these callables based on the tool call identifiers:

```python theme={null}
tools = ["web_search", "calculator"]

function_map = {
    "web_search": web_search_function,
    "calculator": calculator_function
}

agent.action(
    input_text="What is 42 divided by 7?",
    tools=tools,
    function_map=function_map
)
```

***

## 🚀 Guide: Using the `action` Command

The `action` function lets your agent execute tasks in the desktop environment. It handles:

* **Starting a new conversation** with a command.
* **Continuing a conversation** by supplying user input.
* **Acknowledging safety checks** for a pending call.
* **Auto-handling safety checks and input** if `ignore_safety_and_input=True`.
* **Custom handler delegation** for each status.

Internally, `action` manages state and either returns a `(status, data)` tuple for you to process or calls the appropriate handler if provided.

***

### 📌 Quick Overview

```python theme={null}
def action(
    input_text=None,
    acknowledged_safety_checks=False,
    ignore_safety_and_input=False,
    complete_handler=None,
    needs_input_handler=None,
    needs_safety_check_handler=None,
    error_handler=None
):
    # ...
```

* **`input_text`** (str, optional):
  * A new command to start a conversation.
  * A user’s response if the agent has asked for more input.
  * `None` if you’re just confirming safety checks.

* **`acknowledged_safety_checks`** (bool, optional):
  * Indicates that the user has confirmed pending checks.
  * Only relevant if a **NEEDS\_SECURITY\_CHECK** status was returned previously.

* **`ignore_safety_and_input`** (bool, optional):
  * If `True`, the function automatically handles safety checks and input requests, requiring no user interaction.

* **Handlers** (callables, optional):
  * `complete_handler(data)`: Handles **COMPLETE**.
  * `needs_input_handler(messages)`: Handles **NEEDS\_INPUT**.
  * `needs_safety_check_handler(checks, pending_call)`: Handles **NEEDS\_SECURITY\_CHECK**.
  * `error_handler(error_message)`: Handles **ERROR**.

**Return Value**:

* **A tuple** `(status, data)`, where:
  * **status** is one of:
    * **COMPLETE**: The agent finished successfully.
    * **ERROR**: An error occurred.
    * **NEEDS\_INPUT**: The agent needs more user input.
    * **NEEDS\_SECURITY\_CHECK**: The agent needs confirmation for a risky action.
  * **data**:
    * **For COMPLETE**: The final response object.
    * **For ERROR**: An error message.
    * **For NEEDS\_INPUT**: A list of messages asking for input.
    * **For NEEDS\_SECURITY\_CHECK**: A list of safety checks and the pending call.

When handlers are provided, `action` may not return a status in the usual way—it delegates behavior to those handlers.

***

### 🌀 Handling the Workflow (Interactive Example)

`action` covers multiple scenarios:

1. **Starting a conversation** with `input_text` (e.g., a command: “Open Chrome”).
2. **Continuing a conversation** by providing user input if the agent is waiting for it.
3. **Acknowledging safety checks** with `acknowledged_safety_checks=True` if the agent flagged a security concern.
4. **Auto-handling** if `ignore_safety_and_input=True`, which bypasses user checks.

When you call `action`, you get `(status, data)` back (unless you use handlers). Use `status` to decide your next move:

* **COMPLETE**:
  * The task is done. `data` contains the final response.
  * You can display or log it.

* **ERROR**:
  * `data` holds an error message explaining what went wrong.
  * You can retry, log, or show it to the user.

* **NEEDS\_INPUT**:
  * The agent requires more information. Use `data` (often a list of prompts) to know what it wants.
  * Get input from the user, then call `action` again, **supplying that text in `input_text`**.

* **NEEDS\_SECURITY\_CHECK**:
  * The agent found a risky action. You must confirm it’s safe.
  * Call `action` again with **`acknowledged_safety_checks=True`** to proceed.
  * No extra `input_text` is required unless the agent specifically requests it.

Here’s a straightforward example:

```python theme={null}
status, data = agent.action(input_text="Open Firefox")

if status == AgentStatus.COMPLETE:
    print("Done:", data)
elif status == AgentStatus.ERROR:
    print("Error:", data)
elif status == AgentStatus.NEEDS_INPUT:
    user_reply = input(f"Input needed: {data}")
    agent.action(input_text=user_reply)
elif status == AgentStatus.NEEDS_SECURITY_CHECK:
    confirm = input(f"Security checks: {data['safety_checks']} Proceed? (y/N): ")
    if confirm.lower() == "y":
        agent.action(acknowledged_safety_checks=True)
    else:
        print("Action cancelled.")
```

For a more robust loop-based approach:

```python theme={null}
status, data = agent.action(input_text="Open a file")

while status in [AgentStatus.NEEDS_INPUT, AgentStatus.NEEDS_SECURITY_CHECK]:
    if status == AgentStatus.NEEDS_INPUT:
        user_reply = input(f"Agent needs more info: {data}")
        status, data = agent.action(input_text=user_reply)
    elif status == AgentStatus.NEEDS_SECURITY_CHECK:
        confirm = input(f"Security checks: {data['safety_checks']} Proceed? (y/N): ")
        if confirm.lower() == "y":
            status, data = agent.action(acknowledged_safety_checks=True)
        else:
            print("Action cancelled.")
            break

if status == AgentStatus.COMPLETE:
    print("Final result:", data)
elif status == AgentStatus.ERROR:
    print("Error:", data)
```

***

### 🤖 Automated (Non-Interactive) Mode

Set **`ignore_safety_and_input=True`** to:

* Automatically approve safety checks.
* Automatically generate responses to agent questions to continue with the prompt

This is useful for:

* Automated actions that must run without user interaction.
* Headless or server-based scenarios.

**CAUTION:** It is inherently risky because **you skip all manual confirmations and user input**. Ensure your applications and use cases are safe before using auto mode.

Example:

```python theme={null}
status, data = agent.action(
    input_text="Open Chrome",
    ignore_safety_and_input=True
)

if status == AgentStatus.COMPLETE:
    print("Completed:", data)
elif status == AgentStatus.ERROR:
    print("Error:", data)
```

***

### Examples Using Handlers

You can avoid manual `if/else` checks by supplying **handlers** for each status. `action` will call them automatically:

* **`complete_handler(data)`**: Called when the agent finishes.
* **`needs_input_handler(messages)`**: Called if the agent wants more input.
* **`needs_safety_check_handler(safety_checks, pending_call)`**: Called if the agent flags a safety check.
* **`error_handler(error_message)`**: Called if something goes wrong.

#### Why Handlers?

* They allow **complex logic** in a more organized, modular style.
* They help you **integrate** with other tools or services, since each status is handled by a dedicated function.
* They reduce repeated conditional code in your main flow.

Example:

```python theme={null}
result = [None]  # a mutable container to store final output or None

def complete_handler(data):
    """COMPLETE -- handle final results"""
    print("\n✅ Task completed successfully!")
    result[0] = data

def needs_input_handler(messages):
    """NEEDS_INPUT -- prompt the user and return the response"""
    for msg in messages:
        if hasattr(msg, "content"):
            text_parts = [part.text for part in msg.content if hasattr(part, "text")]
            print(f"\n💬 Agent asks: {' '.join(text_parts)}")

    user_says = input("Enter your response (or 'exit'/'quit'): ").strip()
    if user_says.lower() in ("exit", "quit"):
        print("Exiting as per user request.")
        result[0] = None
        return None
    return user_says

def needs_safety_check_handler(safety_checks, pending_call):
    """NEEDS_SAFETY_CHECK -- confirm or deny safety checks"""
    for check in safety_checks:
        if hasattr(check, "message"):
            print(f"☢️  Pending Safety Check: {check.message}")

    ack = input("Type 'ack' to confirm, or 'exit'/'quit': ").strip().lower()
    if ack in ("exit", "quit"):
        print("Exiting as per user request.")
        result[0] = None
        return False
    if ack == "ack":
        print("Acknowledged. Proceeding with the computer call...")
        return True
    return False

def error_handler(error_message):
    """ERROR -- print error and store None"""
    print(f"😱 ERROR: {error_message}")
    result[0] = None

# Provide handlers to `action`:
status, data = desktop.action(
    input_text="Open Chrome",
    complete_handler=complete_handler,
    needs_input_handler=needs_input_handler,
    needs_safety_check_handler=needs_safety_check_handler,
    error_handler=error_handler
)
```

When handlers are specified, `action` manages each status internally and continues until it hits **COMPLETE** or **ERROR** (unless you stop it prematurely).

***

### 4. Key Takeaways

1. **Scenarios**: Start new tasks, resume with user input, or confirm safety checks.
2. **Statuses**: Always handle COMPLETE, ERROR, NEEDS\_INPUT, NEEDS\_SECURITY\_CHECK.
3. **Resuming**: Pass new input (`input_text`) or confirm checks (`acknowledged_safety_checks=True`) to continue.
4. **Auto-mode**: `ignore_safety_and_input=True` is convenient but risky.
5. **Handlers**: Offer a cleaner, more modular way to manage status-based logic.

> For any additional questions, contact [founders@passage-team.com](mailto:founders@passage-team.com)

***