PromptOps: LLM-Powered Desktop Automation
PromptOps is a local-first LLM automation system that mimics how humans operate a computer—via reasoning, vision, and keystrokes. It interprets natural language goals and executes them like a real user would, using keyboard inputs and visual feedback to interact with applications.

PromptOps: LLM-Powered Desktop Automation
PromptOps performs desktop automation by leveraging the power of Large Language Models (LLMs) to understand and execute natural language instructions. This project aims to create a flexible and robust system capable of automating complex tasks without relying on hardcoded scripts or application-specific APIs.
Overview
PromptOps takes natural language prompts as input, translates them into a sequence of keyboard actions, and executes them on the desktop. The system utilizes computer vision to understand the current screen state and dynamically adapt its actions based on the observed output. This allows for a more robust and adaptable automation system than traditional methods.
Key Features
- Natural Language Understanding via LLMs
- Keyboard-driven UI interaction
- Vision-based screen content interpretation
- Adaptive planning based on execution outcomes
- Modular design for easy extensibility
Technical Implementation
The core logic is implemented in Python, utilizing PyAutoGUI/Pynput for keyboard control and an LLM (e.g., Gemini) for reasoning and vision analysis. The system is designed with a modular architecture, encompassing a planner, an executor, an evaluator, and a vision agent. Each module interacts with a central controller to ensure coordinated execution. FastAPI provides optional API hooks for integration with other systems.
Challenges & Solutions
Challenges included achieving reliable UI control , robustly parsing dynamic screen content, and balancing flexibility with deterministic execution. Solutions involved implementing advanced image recognition techniques, developing context-aware parsing algorithms, and designing a flexible planner that can handle unexpected situations.
The real power of PromptOps lies in its ability to adapt and learn, making it a truly intelligent automation system.
See more
More projects

GitHub Wiki Vault: A Brain Map of My Code
An Obsidian vault that maps every repository on my GitHub account from its source code, grown every night by an AI agent pipeline that I designed and review. The code is private on purpose.

Tally: Bank Statements to Answers
A personal finance app for Indian bank and UPI statements. It parses, de-duplicates and categorises every transaction, then answers questions about your spending in plain English.

VectorSwift: Visual Pipeline Builder
Build, connect, and validate data pipelines visually with VectorSwift, a React-based DAG editor featuring real-time validation and backend integration.