Exploring the Potential of Computer-Using Agents
This post was originally published on LinkedIn.
At Avanade, we continuously explore emerging technologies and their practical applications. Recently, across the Office of the CTO, we have been experimenting with ChatGPT Operator and computer-using agents. We sat down with our experts Chris Lloyd-Jones (Head of Platforms and Incubation, Office of the CTO), Diana Wolfe (Lead Researcher, R&D, Office of the CTO), and Fergus Kidd (Engineering Lead, R&D, Office of the CTO) to discuss their experiences and insights from months of testing these intelligent agents.
First Impressions: Excitement Meets Reality
Initially, our team approached ChatGPT Operator with high expectations. Chris found immediate utility in tasks that traditional search engines handled poorly, like navigating booking sites or extracting structured data. Fergus, on the other hand, was impressed by initial capabilities but soon encountered practical limitations, especially when the tasks required significant human oversight. Diana experienced an opposite trajectory, initially disappointed but later found niche tasks where Operator excelled, once its limitations were clear.
Real-world Use Cases: The Good, the Bad, and the Unexpected
Chris shared practical examples ranging from personal tasks, like selecting greeting cards from Moonpig, to professional ones such as assessing FSA coverage or evaluating meetup groups. He highlighted that Operator was most effective for sites where content was structured in tables and less accessible to traditional search engines.
Fergus explored repeatable administrative tasks like managing Azure subscriptions. However, he often concluded tasks could be quicker or simpler via existing APIs or direct coding. Notably, Fergus experimented with data extraction tasks, such as compiling addresses from FedEx locations, finding Operator capable but inconsistent, highlighting the necessity of human verification.
Diana illustrated the limitations with a travel-planning scenario, where Operator struggled without detailed guidance, demonstrating it didn’t naturally request crucial contextual information like hotel locations. Despite this, Diana recognized Operator’s potential in specific contexts like budget-constrained flight bookings.
Limitations and Challenges
Several technical hurdles became apparent. The team consistently noted difficulties with handling CAPTCHA challenges, limited understanding of nuanced interfaces, and constraints on secure systems—like corporate webmail, which requires trusted certificates inaccessible by external Operator agents.
Chris pointed out that, contrary to initial expectations rooted in robotic process automation (RPA), the agent struggled with UI interactions. Diana echoed this, expecting more robust interactivity based on prior experiences with similar technologies.
Future Opportunities: From Testing to Practical Integration
Despite its current limitations, our team sees significant potential for computer-using agents, particularly in roles involving repetitive tasks and user experience testing. Fergus and Diana suggested customer service, data entry, and accessibility auditing as promising areas. Chris expanded further, envisioning agents as effective testers for interfaces, mimicking diverse user behaviors to identify unexpected pathways and usability issues.
A fascinating example Diana shared was the experimental use of Claude, an agent attempting to play Pokémon. Although currently limited, such interactions demonstrate the potential future capability of agents to operate autonomously in complex, dynamic environments.
Looking Ahead
Our consensus is clear: the core technology behind computer-using agents holds enormous promise. The critical next step is tighter integration into existing systems and platforms, allowing precise training and tailored deployments. Fergus emphasized the need for increased control and adaptability, and Diana underscored embedding these agents into real workflows for tangible efficiency gains.
At Avanade, we remain excited about the evolution of these technologies. As we continue our exploration, we look forward to sharing more insights into harnessing computer-using agents effectively, making everyday tasks simpler, smarter, and more efficient.
Stay tuned for further updates from our Office of the CTO team!