Check out the announcement. GPT-4 accepts mixed-mode inputs of text and images.

Mouse cursor instructions aren’t a massive leap from the current capabilities, given the rate of progress and recent developments around LLM tool use and the like.