Introduction

iauto is a low-code engine for building and deploying AI agents.

• AI: Cutting-edge AI capabilities.
• Low-Code: Define your agents using YAML.
• Automation: Integrated automation frameworks like Appium and Playwright.
• Extensible: Well-designed Python API.

News

• Integrated autogen: Create your Multi-Agent system efficiently with iauto.
• Run most open source LLMs locally using llama.cpp and llama-cpp-python.

Installation

Python version requirement: >=3.8

iauto can be installed from PyPI using pip. It is recommended to create a new virtual environment before installation to avoid conflicts.

pip install -U iauto

To access the latest code, install directly from the GitHub repository.

pip install git+https://github.com/shellc/iauto.git

Enable hardware acceleration for local LLM

If you want to run LLM locally, you can enable hardware acceleration in the following ways.

To enable cuBLAS acceleration on NVIDIA GPU:

CMAKE_ARGS="-DGGML_CUBLAS=ON" pip install -U iauto

To enable Metal on Apple silicon devices:

CMAKE_ARGS="-DGGML_METAL=ON" pip install -U iauto

Quick start

Browser recording to AI replay

More playbooks for AI

LLM possesses a robust natural language understanding capability, allowing AI systems built with LLM to grasp human intent effectively. This revolutionizes the way individuals interact with software, as they can now give commands using natural language for software to perform specific tasks. By harnessing LLM's reasoning capacity, more intelligent Agent systems can be created. These Agents not only understand human intent to carry out direct commands but also analyze, break down, and plan task execution for more intricate tasks. This enables humans to delegate higher-level tasks instead of being restricted to specific instructions.

There is a growing demand for web automation, typically achieved through RPA. The potential for AI to handle web automation tasks is promising, allowing for more intuitive and efficient execution of tasks. The capabilities of AI in web automation rely on the functions provided by programmers, highlighting the importance of a well-rounded set of functions for optimal performance.

Chrome browser has a built-in feature that allows users to record their actions on a webpage and then replay them at a later time. This feature is called "Record and Replay" and it is designed to help developers and testers easily create and replay test scripts.

Enabling AI to replay the results of Browser recording as needed would greatly enhance AI's capabilities for web automation.

iauto is an automation toolset designed for AI, featuring workflow orchestration, automation, and AI Agent capabilities. It allows users to orchestrate task flows through low-code playbooks, which can be utilized as tools by the AI system on demand.

I will show you how to use browser recording to create playbooks for AI, step by step.

Browser recording

We will create a function for Google flight search that can be utilized by AI.

1. Open your Chrome web browser, go to https://www.google.com/travel/flights, and press F12 to access DevTools.
2. Select Recorder tool.
3. Click Create a new recording .
4. Click Start recording to begin recording.
5. In the browser, choose one-way, enter the departure and arrival locations and date, then click search.
6. Click End recording to end recording.
7. Click on Replay to check if the recorded process functions correctly. 8 Click Export or Show code to export the recorded operations as a JSON file.

Now that I have obtained a JSON file recording the operation process, I will save it as google_flights_search_replay_script.json.

We will replace some input data recorded during the recording process with variables so that AI can substitute these values.

{
  "title": "Recording 2/28/2024 at 1:05:30 PM",
  "steps": [
    {
      "type": "setViewport",
      "width": 1369,
      "height": 1213,
      "deviceScaleFactor": 1,
      "isMobile": false,
      "hasTouch": false,
      "isLandscape": false
    },
    {
      "type": "navigate",
      "url": "https://www.google.com/travel/flights",
      "assertedEvents": [
        {
          "type": "navigation",
          "url": "https://www.google.com/travel/flights",
          "title": "Google Flights - Find Cheap Flight Options & Track Prices"
        }
      ]
    },
    {
      "type": "click",
      "target": "main",
      "selectors": [
        ["aria/Change ticket type. ​Round trip"],
        ["div.RLVa8 div.VfPpkd-TkwUic"],
        [
          "xpath///*[@id=\"yDmH0d\"]/c-wiz[2]/div/div[2]/c-wiz/div[1]/c-wiz/div[2]/div[1]/div[1]/div[1]/div/div[1]/div[1]/div/div/div/div[1]"
        ],
        ["pierce/div.RLVa8 div.VfPpkd-TkwUic"]
      ],
      "offsetY": 21,
      "offsetX": 56.5
    },
    {
      "type": "click",
      "target": "main",
      "selectors": [
        ["aria/One way"],
        ["div.RLVa8 li:nth-of-type(2)"],
        [
          "xpath///*[@id=\"yDmH0d\"]/c-wiz[2]/div/div[2]/c-wiz/div[1]/c-wiz/div[2]/div[1]/div[1]/div[1]/div/div[1]/div[1]/div/div/div/div[2]/ul/li[2]"
        ],
        ["pierce/div.RLVa8 li:nth-of-type(2)"]
      ],
      "offsetY": 40,
      "offsetX": 75.5
    },
    {
      "type": "click",
      "target": "main",
      "selectors": [
        ["aria/Where from?"],
        ["div.BGeFcf div.cQnuXe input"],
        ["xpath///*[@id=\"i21\"]/div[1]/div/div/div[1]/div/div/input"],
        ["pierce/div.BGeFcf div.cQnuXe input"]
      ],
      "offsetY": 20,
      "offsetX": 70.5
    },
    {
      "type": "change",
      "value": "{$from}",
      "selectors": [
        ["aria/Where else?"],
        ["#i21 > div.ZGEB9c input"],
        ["xpath///*[@id=\"i21\"]/div[6]/div[2]/div[2]/div[1]/div/input"],
        ["pierce/#i21 > div.ZGEB9c input"]
      ],
      "target": "main"
    },
    {
      "type": "keyDown",
      "target": "main",
      "key": "Enter"
    },
    {
      "type": "keyUp",
      "key": "Enter",
      "target": "main"
    },
    {
      "type": "keyDown",
      "target": "main",
      "key": "Tab"
    },
    {
      "type": "keyUp",
      "key": "Tab",
      "target": "main"
    },
    {
      "type": "change",
      "value": "{$to}",
      "selectors": [
        ["aria/Where to? "],
        ["div.vxNK6d div.cQnuXe input"],
        ["xpath///*[@id=\"i21\"]/div[4]/div/div/div[1]/div/div/input"],
        ["pierce/div.vxNK6d div.cQnuXe input"]
      ],
      "target": "main"
    },
    {
      "type": "keyDown",
      "target": "main",
      "key": "Enter"
    },
    {
      "type": "keyUp",
      "key": "Enter",
      "target": "main"
    },
    {
      "type": "keyDown",
      "target": "main",
      "key": "Tab"
    },
    {
      "type": "keyUp",
      "key": "Tab",
      "target": "main"
    },
    {
      "type": "change",
      "target": "main",
      "selectors": [
        "div.ZGEB9c div.kStSsc input",
        "xpath///*[@id=\"ow83\"]/div[2]/div/div[2]/div[1]/div[1]/div[1]/div/input",
        "pierce/div.ZGEB9c div.kStSsc input"
      ],
      "value": "{$date}"
    },
    {
      "type": "keyDown",
      "target": "main",
      "key": "Enter"
    },
    {
      "type": "keyUp",
      "key": "Enter",
      "target": "main"
    },
    {
      "type": "waitForElement",
      "selectors": ["pierce/div.akjk5c span"]
    },
    {
      "type": "click",
      "timeout": 10000,
      "target": "main",
      "selectors": [
        "div.akjk5c span",
        "xpath///*[@id=\"ow83\"]/div[2]/div/div[3]/div[3]/div/button/span",
        "pierce/div.akjk5c span"
      ],
      "offsetX": 9.484375,
      "offsetY": 8,
      "duration": 3000
    },
    {
      "type": "click",
      "target": "main",
      "selectors": [
        "div.MXvFbd span.VfPpkd-vQzf8d",
        "xpath///*[@id=\"yDmH0d\"]/c-wiz[2]/div/div[2]/c-wiz/div[1]/c-wiz/div[2]/div[1]/div[1]/div[2]/div/button/span[2]",
        "pierce/div.MXvFbd span.VfPpkd-vQzf8d",
        "text/Search"
      ],
      "offsetX": 21.453125,
      "offsetY": 11,
      "duration": 1000
    }
  ]
}

Prepare playbook for AI

Next, we will create a playbook that allows AI to be called as a function.

The spec section is used to define the name, description, and arguments details of the function.

The browser.replay is used to replay the process we just recorded.

Use browser.content to retrieve search results.

google_flights_search.yaml:

playbook:
  spec:
    name: google_flights_search
    description: Google flights search.
    arguments:
      - name: from
        type: string
        description: Departure
        required: true
      - name: to
        type: string
        description: Destination
        required: true
      - name: date
        type: string
        description: "Departure, format like: Apr 1"
        required: true
  actions:
    - browser.open:
        args:
          exec: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
          headless: false
          slow_mo: 1000
        result: $browser
    - browser.replay:
        args:
          browser: $browser
          script: ./google_flights_search_replay_script.json
    - time.wait: 3
    - browser.content:
        args:
          browser: $browser
          selector: ".pIav2d .JMc5Xc"
        result: $text
    - browser.close: $browser
    - echo: $text

You can now hand over this playbook to AI, which will automatically execute it when needed.

We reused the content from the playbooks directory here.

llm_chat.yaml:

playbook:
  actions:
    - llm.session:
        args:
          provider: $llm_provider
          llm_args: $llm_args
          tools:
            - shell.cmd
        actions:
          - playbook:
              args:
                - ./google_flights_search.yaml
                - ./bing.yaml
                - ./fetch_links_from_url.yaml
                - ./get_readability_text_from_url.yaml
        result: $session

Play with AI

Let's now launch the playground to test it out. We have selected the ReAct mode, allowing AI to make decisions independently.

We've requested AI to search for flights from San Francisco to Beijing on May 2nd.

It works perfectly!

By leveraging browser recording and iauto's replay capabilities, we can easily integrate web automation with LLM and Agent, enabling us to instruct AI to perform more complex tasks through conversations. This approach enhances work efficiency and automation levels.

iauto Python API Reference

• iauto
  • VERSION
• iauto.log
  • get_level
  • get_logger
  • logger
• iauto.agents
• iauto.agents.executor
  • AgentExecutor
• iauto.actions.playbook
  • Playbook
  • from_dict
  • load
• iauto.actions
• iauto.actions.action
  • ActionArg
  • ActionSpec
  • Action
  • FunctionAction
  • create
• iauto.actions.loader
  • ActionLoader
  • register
• iauto.actions.executor
  • Executor
  • PlaybookExecutor
  • execute
• iauto.llms.chatglm
  • ChatGLM
• iauto.llms.session
  • Session
• iauto.llms.llm
  • Function
  • ToolCall
  • Usage
  • ChatMessage
  • LLM
• iauto.llms.llama
  • LLaMA
• iauto.llms.openai
  • OpenAI
• iauto.llms.llm_factory
  • create_llm

iauto

iauto is a Low-Code intelligent automation tool that integrates LLM and RPA.

Classes:

• Playbook
• PlaybookExecutor

VERSION

The current version.

iauto.log

get_level

def get_level(name: str)

Get the log level from string.

Arguments:

• name str - Log level string.

Returns:

• int - The corresponding log level.

get_logger

def get_logger(name, level=None)

Get a logger with the given name and level.

Arguments:

• name str - The name of the logger.
• level int, optional - The logging level. Defaults to None.

Returns:

• logging.Logger - A logger with the specified name and level.

logger

Default logger.

iauto.agents

This module provides the AgentExecutor class which is responsible for executing actions on behalf of agents. It acts as an intermediary layer between the agent's instructions and the actual execution of those instructions.

Classes:

• AgentExecutor

iauto.agents.executor

AgentExecutor Objects

class AgentExecutor()

run

def run(message: ChatMessage,
        clear_history: Optional[bool] = True,
        silent: Optional[bool] = False,
        **kwargs) -> Dict

Runs the chat session with the given message and configuration.

This method initiates a chat with the recipient (either a single agent or a group chat manager) using the UserProxyAgent. It processes the message, manages the chat history, and generates a summary of the conversation.

Arguments:

• message ChatMessage - The message to start the chat with.
• clear_history Optional[bool] - Determines whether to clear the chat history before starting the new chat session. Defaults to True.
• silent Optional[bool] - If set to True, the agents will not output any messages. Defaults to False.
• **kwargs - Additional keyword arguments that might be needed for extended functionality.

Returns:

• Dict - A dictionary containing the chat history, summary of the conversation, and the cost of the session.

reset

def reset()

Resets the state of all agents and the UserProxyAgent.

This method clears any stored state or history in the agents to prepare for a new task.

set_human_input_mode

def set_human_input_mode(mode)

Sets the human input mode for the UserProxyAgent and the recipient.

Arguments:

• mode str - The mode of human input to set. Can be 'NEVER', 'TERMINATE', or 'ALWAYS'.

register_human_input_func

def register_human_input_func(func)

Registers a function to handle human input across all agents.

Arguments:

• func Callable - The function to be called when human input is needed.

register_print_received

def register_print_received(func)

Registers a function to print messages received by agents.

The function will be called each time an agent receives a message, unless the silent flag is set to True.

Arguments:

• func Callable - The function to be called with the message, sender, and receiver information when a message is received.

iauto.actions.playbook

Playbook Objects

class Playbook(BaseModel)

A class representing a playbook which includes a series of actions to be executed.

Attributes:

• name Optional[str] - The name of the playbook.
• description Optional[str] - A brief description of what the playbook does.
• args Union[str, List, Dict, None] - Arguments that can be passed to the actions in the playbook.
• actions Optional[List['Playbook']] - A list of actions (playbooks) to be executed.
• result Union[str, List, Dict, None] - The result of the playbook execution.

resolve_path

def resolve_path(path: str) -> str

Resolves a potentially relative path to an absolute path using the playbook's metadata.

Arguments:

• path str - The file path that may be relative or absolute.

Returns:

• str - The absolute path resolved from the given path and the playbook's metadata root.

from_dict

def from_dict(d: Dict) -> Playbook

Generate a Playbook object from the input dictionary.

Attributes:

• d Dict - The dictionary to convert into a Playbook object.

Returns:

• playbook Playbook - The converted Playbook object.

load

def load(fname: str) -> Playbook

Load a playbook from file.

Attributes:

• fname str - Path to the YAML file containing the playbook.

Returns:

• playbook Playbook - The loaded playbook.

iauto.actions

This module defines the core components of the action system.

It provides the necessary classes and functions to create, manage, and execute actions within the context of the playbook execution environment. It includes definitions for actions, action arguments, action specifications, executors, and playbooks.

Classes:

• Action: Represents a single action to be executed.
• ActionArg: Defines an argument that can be passed to an action.
• ActionSpec: Contains the specification details of an action.
• Executor: The base class for action executors.
• Playbook: Represents a sequence of actions to be executed as a unit.
• PlaybookExecutor: Responsible for executing the actions defined in a playbook.
• PlaybookRunAction: A special action that represents the execution of a playbook.

Functions:

• create_action: A factory function to create instances of actions.
• loader: A function to load actions and their specifications.
• register_action: A function to register new actions into the system.

The module also handles the registration and discovery of built-in actions.

iauto.actions.action

ActionArg Objects

class ActionArg(BaseModel)

A class representing an argument for an action.

Attributes:

• name str - The name of the argument.
• type str - The type of the argument, default is "string".
• description str - A description of what the argument is for.
• required bool - Whether the argument is required or optional, default is False.

ActionSpec Objects

class ActionSpec(BaseModel)

A class representing the specification of an action.

Attributes:

• name str - The name of the action.
• description str - A brief description of what the action does.
• arguments Optional[List[ActionArg]] - A list of arguments that the action accepts.

from_dict

@staticmethod
def from_dict(d: Dict = {}) -> 'ActionSpec'

Create an ActionSpec instance from a dictionary representation.

Arguments:

• d Dict, optional - The dictionary containing action specification data.

Returns:

• ActionSpec - An instance of ActionSpec created from the provided dictionary.

Raises:

• ValueError - If the dictionary contains invalid data for creating an ActionSpec.

from_oai_dict

@staticmethod
def from_oai_dict(d: Dict = {}) -> 'ActionSpec'

Create an ActionSpec instance from a dictionary following the OpenAPI Specification.

Arguments:

• d Dict, optional - The dictionary containing OpenAPI Specification data.

Returns:

• ActionSpec - An instance of ActionSpec created from the provided OpenAPI dictionary.

Raises:

• ValueError - If the dictionary does not conform to the expected OpenAPI Specification format.

oai_spec

def oai_spec() -> Dict

Generate an OpenAPI Specification dictionary for this action.

Returns:

• Dict - A dictionary representing the action in OpenAPI Specification format.

Action Objects

class Action(ABC)

Abstract base class for an action.

An action defines a single operation that can be performed. Actions are typically executed by an Executor within the context of a Playbook.

Attributes:

• spec ActionSpec - The specification of the action.

perform

@abstractmethod
def perform(*args, **kwargs) -> Any

Execute the action with the given arguments.

Arguments:

• *args - Positional arguments for the action.
• **kwargs - Keyword arguments for the action.

Returns:

• Any - The result of the action execution.

Raises:

• NotImplementedError - If the method is not implemented by the subclass.

copy

def copy()

Create a copy of the Action instance.

Returns:

• Action - A new instance of the Action with the same specification.

FunctionAction Objects

class FunctionAction(Action)

A concrete implementation of Action that wraps a Python callable.

Attributes:

• _func Callable - The Python callable to wrap.
• spec ActionSpec - The specification of the action.

perform

def perform(*args, **kwargs) -> Any

Execute the wrapped Python callable with the given arguments.

Arguments:

• *args - Positional arguments for the callable.
• **kwargs - Keyword arguments for the callable.

Returns:

• Any - The result of the callable execution.

create

def create(func, spec: Dict) -> Action

Factory function to create a FunctionAction.

Arguments:

• func Callable - The Python callable that the action will execute.
• spec Dict - A dictionary representing the action's specification.

Returns:

• Action - A FunctionAction instance with the given callable and specification.

iauto.actions.loader

ActionLoader Objects

class ActionLoader()

Manages the registration and retrieval of action instances.

This class provides a mechanism to register actions by name and retrieve them. It keeps an internal dictionary that maps action names to action instances.

register

def register(actions: Dict[str, Action])

Registers a set of actions.

Arguments:

• actions Dict[str, Action] - A dictionary with action names as keys and Action instances as values to be registered.

get

def get(name) -> Union[Action, None]

Retrieves an action instance by its name.

Arguments:

• name str - The name of the action to retrieve.

Returns:

Action or None: The action instance if found, otherwise None.

actions

@property
def actions()

Gets a list of all registered action instances.

Returns:

• list - A list of Action instances.

load

def load(identifier)

Loads an action from a given identifier.

The identifier is expected to be a string in the format 'package.module.ClassName', where 'package.module' is the full path to the module containing the class 'ClassName' that is the action to be loaded.

Arguments:

• identifier str - A dot-separated path representing the action to be loaded.

Raises:

• ValueError - If the action name conflicts with an already registered action.
• ImportError - If the module cannot be imported.
• AttributeError - If the class cannot be found in the module.

Returns:

None

register

def register(name: str, spec: Dict)

Registers a new action with the provided name and specification.

Arguments:

• name str - The unique name of the action to register.
• spec Dict - A dictionary containing the action specification.

Returns:

The created action instance.

Decorates:

• func - The function to be transformed into an action.

Raises:

• ValueError - If an action with the given name already exists.

iauto.actions.executor

Executor Objects

class Executor(ABC)

Abstract base class for an executor that can run playbooks.

perform

@abstractmethod
def perform(playbook: Playbook) -> Any

Execute the given playbook.

Arguments:

• playbook Playbook - The playbook to execute.

Returns:

• Any - The result of the playbook execution.

get_action

@abstractmethod
def get_action(playbook: Playbook) -> Union['Action', None]

Retrieve the action associated with the given playbook.

Arguments:

• playbook Playbook - The playbook containing the action to retrieve.

Returns:

Union[Action, None]: The action to perform, or None if not found.

eval_args

@abstractmethod
def eval_args(args: Union[str, List, Dict, None] = None) -> Tuple[List, Dict]

Evaluate the arguments passed to the playbook or action.

Arguments:

• args Union[str, List, Dict, None], optional - The arguments to evaluate.

Returns:

Tuple[List, Dict]: A tuple containing the evaluated arguments as a list or a dictionary.

extract_vars

@abstractmethod
def extract_vars(data, vars)

Extract variables from the given data based on the vars specification.

Arguments:

• data - The data from which to extract variables.
• vars - The specification of variables to extract from the data.

set_variable

def set_variable(name: str, value: Any)

Set a variable in the executor's context.

Arguments:

• name str - The name of the variable to set.
• value Any - The value to assign to the variable.

variables

@property
def variables() -> Dict

Get the current variables in the executor's context.

Returns:

• Dict - A dictionary of the current variables.

PlaybookExecutor Objects

class PlaybookExecutor(Executor)

Executes playbooks containing a sequence of actions.

This executor handles the running of actions defined in a playbook, managing the necessary thread execution for asynchronous actions and the extraction and evaluation of variables from the results.

perform

def perform(playbook: Playbook) -> Any

Executes the given playbook.

This method takes a Playbook object, retrieves the corresponding action, evaluates the arguments, and then performs the action. If the action's result is an asyncio coroutine, it is run in a separate thread. After the action is performed, any variables specified in the playbook's 'result' section are extracted and stored.

Arguments:

• playbook (Playbook): The playbook object containing the action to be executed.

Returns:

• Any: The result of executing the action in the playbook.

Raises:

• ValueError: If the action is not found or the playbook is invalid.

eval_vars

def eval_vars(vars)

Evaluate the variables in the context of the current executor's state.

This method takes a variable or a structure containing variables and evaluates them using the current state of variables stored within the executor. It supports strings, lists, and dictionaries. If a string variable starts with '$', it is treated as a reference to a variable which is then resolved. If the variable is not found, the string is returned as is. For lists and dictionaries, each element or key-value pair is recursively processed.

Arguments:

• vars (Union[str, List, Dict, None]): The variable or structure containing variables to be evaluated.

Returns:

• The evaluated variable, or the original structure with all contained variables evaluated.

eval_args

def eval_args(args: Union[str, List, Dict, None] = None) -> Tuple[List, Dict]

Evaluates and separates arguments into positional and keyword arguments.

This method processes the input arguments, evaluating any variables within them using the current state of the executor. It then separates the evaluated arguments into positional (list) and keyword (dictionary) arguments.

Arguments:

• args (Union[str, List, Dict, None], optional): The arguments to evaluate and separate. Can be a string, list, dictionary, or None.

Returns:

• Tuple[List, Dict]: A tuple containing a list of positional arguments and a dictionary of keyword arguments.

extract_vars

def extract_vars(data, vars)

Extracts variables from the result data and stores them in the executor's state.

This method takes the result data and the variables defined in the playbook's 'result' section, then stores these variables in the executor's state for later use. The method supports extracting variables from strings, lists, and dictionaries.

Arguments:

• data (Any): The result data from which variables will be extracted.
• vars (Union[str, List, Dict, None]): The structure defining which variables to extract from the result data. It can be a string, a list, a dictionary, or None.

If vars is a string that starts with '$', it is assumed to be a variable name, and the entire data is stored with that variable name. If vars is a list, each element in vars that starts with '$' is treated as a variable name, and the corresponding element in data is stored with that variable name, provided data is also a list and the indices match. If vars is a dictionary, each key that starts with '$' is treated as a variable name, and the value from data corresponding to the dictionary's value is stored with the variable name, provided data is also a dictionary and contains the keys.

No action is taken if vars is None or if the data types of vars and data do not correspond as expected.

resolve_path

def resolve_path(path: str) -> str

Resolves a given file path to an absolute path.

If the given path is already an absolute path, it is returned unchanged. Otherwise, the path is resolved relative to the directory of the current playbook file, which is stored in the executor's variables under the key "file".

Arguments:

• path str - The file path to resolve.

Returns:

• str - The resolved absolute file path.

execute

def execute(playbook: Union[str, Playbook], variables={}) -> Any

Executes a playbook from a given file with optional initial variables.

This static method loads a playbook from the specified file, creates a new PlaybookExecutor instance, sets any initial variables, and then performs the playbook.

Arguments:

• playbook_file str - The path to the playbook file to be executed.
• variables dict, optional - A dictionary of initial variables to set in the executor's context before executing the playbook. Defaults to an empty dictionary.

Returns:

• Any - The result of executing the playbook, which could be of any type depending on the actions performed within the playbook.

iauto.llms.chatglm

ChatGLM Objects

class ChatGLM(LLM)

generate

def generate(instructions: str, **kwargs) -> Message

iauto.llms.session

Session Objects

class Session()

The Session class is responsible for managing a conversation with a language model (LLM). It handles the state of the conversation, including the messages exchanged and the actions that can be performed based on those messages.

The Session class provides high-level methods to interact with a language model, allowing for complex conversation flows, tool integration, and message management.

add

def add(message: ChatMessage) -> None

Add a new ChatMessage to the session's message history.

Arguments:

• message ChatMessage - The message to add to the history.

Returns:

None

llm

@property
def llm() -> LLM

Get the language model (LLM) instance associated with the session.

Returns:

• LLM - The language model instance used for processing messages.

messages

@property
def messages() -> List[ChatMessage]

Get the list of ChatMessage instances that represent the message history of the session.

Returns:

• List[ChatMessage] - The list of messages exchanged during the session.

actions

@property
def actions() -> Optional[List[Action]]

Get the list of Action instances that can be performed within the session.

Returns:

• Optional[List[Action]] - The list of actions available in the session, or an empty list if none are set.

run

def run(instructions: Optional[str] = None,
        messages: Optional[List[ChatMessage]] = None,
        history: int = 5,
        rewrite: bool = False,
        expect_json: int = 0,
        tools: Optional[List[Action]] = None,
        use_tools: bool = True,
        auto_exec_tools: bool = True,
        **kwargs) -> Union[ChatMessage, Dict, List]

Run a conversation flow based on provided instructions and messages, with the option to rewrite the input, expect a JSON response, and execute tools.

Arguments:

• instructions Optional[str] - Instructions to prepend to the messages before sending to the LLM as a system role message.
• messages Optional[List[ChatMessage]] - A list of ChatMessage instances to include in the conversation. If not provided, the last 'history' number of messages from the session will be used.
• history int - The number of recent messages from the session to consider in the conversation. Defaults to 5.
• rewrite bool - Whether to rewrite the last user message to be clearer before running the session.
• expect_json int - The number of times to attempt parsing the LLM's response as JSON before giving up.
• tools Optional[List[Action]] - A list of Action instances representing tools that can be used in the session.
• use_tools bool - Whether to include tool specifications when sending messages to the LLM.
• auto_exec_tools bool - Automatically execute tools if they are called in the LLM's response.

Returns:

Union[ChatMessage, Dict]: The final ChatMessage from the LLM, or a dictionary if a JSON response is expected and successfully parsed.

Raises:

• json.JSONDecodeError - If 'expect_json' is greater than 0 and the LLM's response cannot be parsed as JSON.

react

def react(instructions: Optional[str] = None,
          messages: Optional[List[ChatMessage]] = None,
          history: int = 5,
          rewrite: bool = False,
          log: bool = False,
          max_steps: int = 3,
          tools: Optional[List[Action]] = None,
          use_tools: bool = True,
          auto_exec_tools: bool = True,
          **kwargs) -> ChatMessage

Perform a series of Thought, Action, and Observation steps to react to the user's last question and generate an answer.

This method simulates a cognitive process by interleaving Thought (reasoning about the situation), Action (using tools to generate an observation or concluding the task), and Observation (result of the action).

Arguments:

• instructions Optional[str] - Additional instructions to provide context for the language model.
• messages Optional[List[ChatMessage]] - The list of ChatMessage instances to include in the conversation. Defaults to the last 'history' messages if not provided.
• history int - The number of recent messages from the session to consider. Defaults to 5.
• rewrite bool - Whether to rewrite the last user message for clarity before processing. Defaults to False.
• log bool - Whether to log the steps of the process. Defaults to False.
• max_steps int - The maximum number of Thought/Action/Observation cycles to perform. Defaults to 3.
• tools Optional[List[Action]] - A list of Action instances representing tools that can be used in the session.
• use_tools bool - Whether to include tool specifications when sending messages to the LLM. Defaults to True.
• auto_exec_tools bool - Whether to automatically execute tools if they are called in the LLM's response. Defaults to True.

Returns:

• ChatMessage - The final ChatMessage containing the answer to the user's question or indicating that more information is needed.

Raises:

• ValueError - If the provided messages list is empty or the last message is not from the user.

rewrite

def rewrite(history: int = 5, **kwargs) -> None

Rewrite the last user message in the session's message history to be clearer and more complete based on the context of the conversation.

This method utilizes the language model to reformulate the user's question, considering the conversation history to provide a clearer version of the question.

Arguments:

• history int - The number of recent messages from the session to consider for context. Defaults to 5.

Returns:

• None - The method updates the last user message in the session's message history in place.

Raises:

• ValueError - If there are no messages or the last message is not from the user.

plain_messages

def plain_messages(messages: List[ChatMessage],
                   norole: bool = False,
                   nowrap: bool = False) -> str

Convert a list of ChatMessage instances into a plain string representation.

Arguments:

• messages List[ChatMessage] - The list of ChatMessage instances to convert.
• norole bool, optional - If True, the role of the message sender is omitted. Defaults to False.
• nowrap bool, optional - If True, newlines in the message content are replaced with '\n'. Defaults to False.

Returns:

• str - A string representation of the messages, each message on a new line.

iauto.llms.llm

Function Objects

class Function(BaseModel)

Represents a function call with optional arguments.

Attributes:

• name str - The name of the function being called.
• arguments Optional[str] - The arguments to be passed to the function, if any.

ToolCall Objects

class ToolCall(BaseModel)

Represents a call to a specific tool with an optional function call.

Attributes:

• id str - The unique identifier for the tool call.
• type str - The type of the tool.
• function Optional[Function] - An optional Function instance representing the function call associated with the tool, if any.

Usage Objects

class Usage(BaseModel)

Represents a token usage.

Attributes:

• input_tokens int - The number of tokens in the input message.
• output_tokens int - The number of tokens in the generated response message.

ChatMessage Objects

class ChatMessage(Message)

Represents a chat message with additional metadata and optional tool call information.

Attributes:

• role str - The role of the entity sending the message (e.g., "user", "system", "assistant").
• tool_calls Optional[List[ToolCall]] - A list of ToolCall instances representing the tool calls associated with this message, if any.
• tool_call_id Optional[str] - The identifier of the tool call associated with this message, if any.
• name Optional[str] - The name of the tool or function called.
• useage Optional[Usage] - The token usage.

from_dict

@staticmethod
def from_dict(d: Dict) -> "ChatMessage"

Create a ChatMessage instance from a dictionary.

Parses the dictionary to populate the ChatMessage fields. If certain keys are not present in the dictionary, default values are used. For 'tool_calls', it creates a list of ToolCall instances from the sub-dictionaries.

Arguments:

• d Dict - The dictionary containing the ChatMessage data.

Returns:

• ChatMessage - An instance of ChatMessage with properties populated from the dictionary.

LLM Objects

class LLM(ABC)

Abstract base class for a Language Model (LLM) that defines the interface for generating messages and handling chat interactions.

generate

@abstractmethod
def generate(instructions: str, **kwargs) -> Message

Generate a message based on the given instructions.

Arguments:

• instructions str - The instructions or prompt to generate the message from.
• **kwargs - Additional keyword arguments that the concrete implementation may use.

Returns:

• Message - The generated message as a Message instance.

chat

@abstractmethod
def chat(messages: List[ChatMessage],
         tools: Optional[List[ActionSpec]] = None,
         **kwargs) -> ChatMessage

Conduct a chat interaction by processing a list of ChatMessage instances and optionally using tools.

Arguments:

• messages List[ChatMessage] - A list of ChatMessage instances representing the conversation history.
• tools Optional[List[ActionSpec]] - An optional list of ActionSpec instances representing tools that can be used in the chat.
• **kwargs - Additional keyword arguments that the concrete implementation may use.

Returns:

• ChatMessage - The response as a ChatMessage instance after processing the interaction.

model

@property
@abstractmethod
def model() -> str

Abstract property that should return the model identifier for the LLM instance.

Returns:

• str - The model identifier.

iauto.llms.llama

LLaMA Objects

class LLaMA(LLM)

llamap.cpp: https://github.com/ggerganov/llama.cpp llama-cpp-python: https://github.com/abetlen/llama-cpp-python

generate

def generate(instructions: str, **kwargs) -> Message

iauto.llms.openai

OpenAI Objects

class OpenAI(LLM)

native_tool_call

def native_tool_call()

Check if the model support fuction calling

iauto.llms.llm_factory

create_llm

def create_llm(provider: str = "openai", **kwargs) -> LLM

Create a language model instance based on the specified provider.

This factory function supports creating instances of different language models by specifying a provider. Currently supported providers are 'openai', 'llama', and 'chatglm'. Depending on the provider, additional keyword arguments may be required or optional.

Arguments:

• provider (str): The name of the provider for the LLM. Defaults to 'openai'.
• **kwargs: Additional keyword arguments specific to the chosen LLM provider.

Returns:

• LLM: An instance of the specified language model.

Raises:

• ImportError: If the required module for the specified provider is not installed.
• ValueError: If an invalid provider name is given.