--- schema: 1 type: video title: >- The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular description: >- A computer agent finds a speaker on Tuesday's conference schedule, saves the successful steps as a skill, and tries to replay them for Wednesday. A hiccup pushes it back into model driven exploration before the workflow completes. Ang Li uses that demonstration to explain Simular's approach to autonomous computers: combine adaptable vision agents with reusable automation code, then check the environment as each step runs. The workshop traces Agent S from accessibility based control to visual grounding, multiple models, and parallel attempts. It also asks a different reliability question from getting one successful result: can the same agent complete the task repeatedly when prompts and environments vary? Li demonstrates a coding agent opening the AI Engineer website, reading an accessibility snapshot, and producing a summary. He then connects code replay and recovery to the tension between stability and adaptation, showing where a fixed script can break and where a model can help repair it. The infrastructure discussion moves from an agent occupying a personal desktop to fleets of cloud computers controlled through an interface or API. Recorded examples include legacy form entry and Android interactions. Audience questions examine screen understanding, verification loops, irreversible actions, operating system restrictions, and workflows built from internal procedures. The session presents an experimental direction with visible limitations: benchmark progress and successful demos still leave substantial work in making everyday computer use reliable. Speaker info: - https://angli.ai/ - https://www.linkedin.com/in/angli-ai - https://github.com/simular-ai/Agent-S Timestamps: 0:00 - Ang and the autonomous computer goal 3:09 - What a computer use agent does 5:03 - Multiple agents and visible desktop work 6:09 - Early visual grounding limitations 7:21 - Autonomy and the computer 9:26 - Screenshots and accessibility signals 10:35 - The Agent S research trajectory 13:34 - Parallel attempts and benchmark progress 15:16 - A FaceTime computer control demo 17:22 - Reliability across repeated runs 19:41 - Clarifying ambiguous instructions 22:12 - The symbolic automation layer 23:22 - Coding agent setup and live demo 25:44 - Write and repair a workflow 26:59 - Read an accessibility snapshot 28:00 - Stability and adaptation 31:07 - Replay code and recover from changes 32:46 - Enable the experimental replay mode 35:12 - Legacy form automation example 37:03 - Find a conference speaker 38:52 - Save the workflow as a skill 39:37 - Replay for a different day 42:10 - Reusing work and reducing model calls 43:34 - Why agents need separate computers 45:09 - Cloud desktops and developer access 47:03 - Multiple platforms and Android demos 50:35 - Developer tools and workshop wrap up 52:10 - Screen understanding and memory questions 53:44 - State and multiple input modalities 57:17 - Accuracy and verification loops 60:30 - Irreversible actions and OS restrictions 63:31 - Legacy workflows and procedures 68:09 - Adapting when applications change 70:22 - Training data and code complexity --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/tyFPqtyOK5w-youtube-8c138db6bc945eee916c Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/tyFPqtyOK5w' markdown: 'https://liquidgold.cc/watch/tyFPqtyOK5w.md' llms: 'https://liquidgold.cc/watch/tyFPqtyOK5w/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:24:27.931Z' language: null children: [] videoId: tyFPqtyOK5w durationSeconds: 4379 publishedAt: '2026-10-11T22:00:02.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 195 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 206 endSeconds: 314 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=206' heldReason: rights - ordinal: 2 text: null startSeconds: 206 endSeconds: 314 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=206' heldReason: rights - ordinal: 3 text: null startSeconds: 383 endSeconds: 448 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=383' heldReason: rights - ordinal: 4 text: null startSeconds: 459 endSeconds: 571 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=459' heldReason: rights - ordinal: 5 text: null startSeconds: 581 endSeconds: 635 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=581' heldReason: rights - ordinal: 6 text: null startSeconds: 646 endSeconds: 817 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=646' heldReason: rights - ordinal: 7 text: null startSeconds: 646 endSeconds: 817 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=646' heldReason: rights - ordinal: 8 text: null startSeconds: 828 endSeconds: 916 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=828' heldReason: rights - ordinal: 9 text: null startSeconds: 828 endSeconds: 916 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=828' heldReason: rights - ordinal: 10 text: null startSeconds: 928 endSeconds: 1050 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=928' heldReason: rights - ordinal: 11 text: null startSeconds: 1061 endSeconds: 1191 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1061' heldReason: rights - ordinal: 12 text: null startSeconds: 1201 endSeconds: 1332 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1201' heldReason: rights - ordinal: 13 text: null startSeconds: 1344 endSeconds: 1402 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1344' heldReason: rights - ordinal: 14 text: null startSeconds: 1416 endSeconds: 1544 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1416' heldReason: rights - ordinal: 15 text: null startSeconds: 1556 endSeconds: 1619 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1556' heldReason: rights - ordinal: 16 text: null startSeconds: 1691 endSeconds: 1867 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1691' heldReason: rights - ordinal: 17 text: null startSeconds: 1877 endSeconds: 1966 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=1877' heldReason: rights - ordinal: 18 text: null startSeconds: 2123 endSeconds: 2223 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2123' heldReason: rights - ordinal: 19 text: null startSeconds: 2237 endSeconds: 2337 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2237' heldReason: rights - ordinal: 20 text: null startSeconds: 2393 endSeconds: 2532 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2393' heldReason: rights - ordinal: 21 text: null startSeconds: 2546 endSeconds: 2622 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2546' heldReason: rights - ordinal: 22 text: null startSeconds: 2634 endSeconds: 2715 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2634' heldReason: rights - ordinal: 23 text: null startSeconds: 2727 endSeconds: 2829 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2727' heldReason: rights - ordinal: 24 text: null startSeconds: 2842 endSeconds: 3046 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=2842' heldReason: rights - ordinal: 25 text: null startSeconds: 3149 endSeconds: 3224 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=3149' heldReason: rights - ordinal: 26 text: null startSeconds: 3235 endSeconds: 3448 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=3235' heldReason: rights - ordinal: 27 text: null startSeconds: 3460 endSeconds: 3630 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=3460' heldReason: rights - ordinal: 28 text: null startSeconds: 3640 endSeconds: 3814 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=3640' heldReason: rights - ordinal: 29 text: null startSeconds: 3826 endSeconds: 4099 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=3826' heldReason: rights - ordinal: 30 text: null startSeconds: 4242 endSeconds: 4379 watch: 'https://liquidgold.cc/watch/tyFPqtyOK5w?t=4242' heldReason: rights --- # The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular A computer agent finds a speaker on Tuesday's conference schedule, saves the successful steps as a skill, and tries to replay them for Wednesday. A hiccup pushes it back into model driven exploration before the workflow completes. Ang Li uses that demonstration to explain Simular's approach to autonomous computers: combine adaptable vision agents with reusable automation code, then check the environment as each step runs. The workshop traces Agent S from accessibility based control to visual grounding, multiple models, and parallel attempts. It also asks a different reliability question from getting one successful result: can the same agent complete the task repeatedly when prompts and environments vary? Li demonstrates a coding agent opening the AI Engineer website, reading an accessibility snapshot, and producing a summary. He then connects code replay and recovery to the tension between stability and adaptation, showing where a fixed script can break and where a model can help repair it. The infrastructure discussion moves from an agent occupying a personal desktop to fleets of cloud computers controlled through an interface or API. Recorded examples include legacy form entry and Android interactions. Audience questions examine screen understanding, verification loops, irreversible actions, operating system restrictions, and workflows built from internal procedures. The session presents an experimental direction with visible limitations: benchmark progress and successful demos still leave substantial work in making everyday computer use reliable. Speaker info: - https://angli.ai/ - https://www.linkedin.com/in/angli-ai - https://github.com/simular-ai/Agent-S Timestamps: 0:00 - Ang and the autonomous computer goal 3:09 - What a computer use agent does 5:03 - Multiple agents and visible desktop work 6:09 - Early visual grounding limitations 7:21 - Autonomy and the computer 9:26 - Screenshots and accessibility signals 10:35 - The Agent S research trajectory 13:34 - Parallel attempts and benchmark progress 15:16 - A FaceTime computer control demo 17:22 - Reliability across repeated runs 19:41 - Clarifying ambiguous instructions 22:12 - The symbolic automation layer 23:22 - Coding agent setup and live demo 25:44 - Write and repair a workflow 26:59 - Read an accessibility snapshot 28:00 - Stability and adaptation 31:07 - Replay code and recover from changes 32:46 - Enable the experimental replay mode 35:12 - Legacy form automation example 37:03 - Find a conference speaker 38:52 - Save the workflow as a skill 39:37 - Replay for a different day 42:10 - Reusing work and reducing model calls 43:34 - Why agents need separate computers 45:09 - Cloud desktops and developer access 47:03 - Multiple platforms and Android demos 50:35 - Developer tools and workshop wrap up 52:10 - Screen understanding and memory questions 53:44 - State and multiple input modalities 57:17 - Accuracy and verification loops 60:30 - Irreversible actions and OS restrictions 63:31 - Legacy workflows and procedures 68:09 - Adapting when applications change 70:22 - Training data and code complexity --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/tyFPqtyOK5w-youtube-8c138db6bc945eee916c Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/tyFPqtyOK5w · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.