Agents submit actions as integers from one to six; action six can encode a click at specified screen coordinates.

“jump out of here what agents will get is agents will get a series of frames and those frames will be 64 by 64 now generally it's just going to be one frame but you might be able to get like maybe two in a row or three in a row and that would show an animation and so beginning state middle state end state and so you can kind of get a perception of over time there what we ask for back is just one through six a series of one through six actions and it's really just integers and so you pass back a one two three four five or six”

“next one two three four five those are just like we call them basic actions and then six could represent a click so like six you submit a six action and a coordinate somewhere on the screen and that represents that you actually clicked on something on there now we we like a really scoped environment because that removes a ton of we'll call it it's really easy to analyze the learning efficiency for humans and ai if we scope down the environment really well so to your point yes it will just be two by two”