Traditional UI automation relies on strict selectors (CSS, XPath, or UI Automation trees). When an application updates its interface, the automation breaks. Generative UI automation solves this.
Computer Vision on Steroids
Using multimodal LLMs (like GPT-4V or Claude 3.5), robots can now “look” at a screen, understand the layout visually, and interact with elements purely by intent. You can instruct the robot to “Click the primary checkout button,” and it will find it, regardless of underlying code changes.
UiPath is leading this charge by integrating advanced computer vision models directly into their core recording tools, making automation significantly more resilient to change.
