Home

Cycle 0

We are building a proof of concept to demonstrate that an LLM can effectively interact with a human UI.

Current Prototype

Our initial prototype operates in a simple loop, making decisions based on deviceType, UI, and the user's goal. While this works for basic tasks, it has several limitations:

• Restricted Actions: It struggles with shortcuts and handling the Enter key.
• Limited Decision-Making: The system is not yet capable of handling complex workflows.

However, we now have a clearer understanding of the problem.

Next Steps

To improve execution and decision-making, we will introduce three brain modules, each with its own specialized context:

1. Task Divider – Breaks down user requests into smaller, manageable tasks.
2. Task Achiever – Executes simple tasks.
3. Task Validator – Verifies whether the Achiever performed correctly.

These modules will communicate with each other to create a more robust system.

---

Cycle 1

Improvements Attempted

To enhance decision-making, I attempted to break down operations further by introducing:

• Planner Agent – Organizes tasks into smaller steps.
• Executor Agent – Performs actions based on the planned steps.

New Challenges

However, this approach introduced new difficulties:

• Debugging Complexity: It's now harder to debug and develop individual agents.
• Prompt Management Issues: Editing prompts and analyzing logs is becoming painful.

Immediate Next Steps

To address these challenges, I need the following improvements:

1. Better Debugging Output:
   • Improve terminal output or create a UI for better visibility.
2. Execution Control:
   • Ability to pause execution, modify the last state’s prompt, and resume from that point.
3. Run Recording & Replay:
   • Record execution runs and use them as a virtual test environment.

By implementing these features, we can make the system more adaptable and easier to iterate on.