Try [a/OmniParser](https://github.com/microsoft/OmniParser) in ComfyUI which a simple screen parsing tool towards pure vision based GUI agent.
https://github.com/smthemex/ComfyUI_OmniParser
git clone https://github.com/smthemex/ComfyUI_OmniParser
comfy node install ComfyUI_OmniParser