We Gave Hermes a Full Toolbox: What Our AI Can Do Now
Today was one of those moments where an AI project stops feeling like a collection of experiments and starts feeling like a real system.
We expanded Hermes, our local AI agent, from a capable assistant into something much closer to a full production operator. The important part is not simply that we turned on more features. The important part is that Hermes can now research, browse, control a computer, work with local files, run code, analyze images and video, generate media, transcribe audio, speak, remember context, delegate tasks, schedule jobs, communicate with other agents, and connect to outside platforms.
That changes what is possible.
Our long-term goal has been simple to describe but difficult to build: give the AI a URL and let the Factory turn it into finished media. Not just a script. Not just a blog. Not just a video. A complete production package. Today moved us much closer to that goal.
The Toolset We Enabled
That is not a small toolset. It is an operating environment. The more interesting question is what an AI can actually do when all of these capabilities are available together.
Research: Web, Browser and X Search
The web tool lets Hermes search the internet, retrieve information and inspect sources. For a Deep Dive project, that can mean starting from one article or URL and expanding it into primary sources, supporting documentation, related reporting, background material and useful visual references.
Browser automation takes that one step further. Searching is passive; browser control is active. Hermes can navigate web apps, click buttons, fill forms, upload files and verify what actually happened on a page. That matters because not every service exposes a clean API. Sometimes the most practical automation is simply operating the same interface a person would use.
X search adds another research channel for fast-moving AI topics, developer reactions, public demos and breaking discussions that may surface before a traditional article is written.
Local Control: Terminal, Files and Code Execution
The terminal remains one of the most important parts of our entire build. Our Factory is deliberately local-first, so Hermes needs to work with PowerShell, Python, FFmpeg, Git, scripts, launchers and local applications.
The file tool gives it access to the project itself: manifests, scripts, audio, subtitles, media, logs, metadata and finished outputs. That is critical because our workflow is not built around isolated prompts. It is built around persistent projects.
Code execution closes the loop. Hermes can do more than suggest code; it can actually run transformations, timing calculations, transcript comparisons, JSON processing, project-state checks and other deterministic work the Factory needs.
Seeing and Hearing: Vision, Video, TTS and STT
Vision lets Hermes inspect thumbnails, screenshots, frames, text placement and user interfaces. Combined with computer use, that means the AI can look at what is happening on-screen and respond to it.
Video analysis is a major upgrade because a successful render is not necessarily a good render. A file can exist and still contain black sections, irrelevant footage, broken captions, missing scenes or a bad ending. Video analysis gives us another independent QA layer after OpenMontage finishes.
Text-to-speech remains central to our content pipeline. A script becomes canonical narration. Then speech-to-text can turn that narration back into timed transcript data and subtitles. That gives us the timing backbone for video production instead of making the editor guess scene lengths.
Media Creation: Image Gen and Video Gen
Image generation can feed blog heroes, thumbnails, editorial cartoons and scene assets directly into the workflow. The value is not simply making a picture; it is making the right picture at the right production stage.
Video generation is now exposed as well. That does not automatically mean free local video generation is configured, and we still need to verify which backend it uses and whether there is any cost. But architecturally, Hermes can now call that class of capability when the Factory needs custom footage.
Skills, Todo, Memory and Context Engine
Skills let us teach Hermes repeatable operating procedures instead of re-explaining every workflow in one enormous prompt. Over time, those skills can become things like research-topic, create-deep-dive-blog, build-youtube-package, produce-openmontage-video, verify-video, publish-blogger and publish-facebook.
Todo gives the project an explicit task layer. A production run can have dozens of steps, and Hermes needs to know what is complete, what failed and what remains.
Memory and session search let the system reuse knowledge from prior work. The context engine is especially interesting for long-running jobs because research, scripts, media decisions, logs, errors and publishing results can quickly exceed what one conversational context can comfortably hold.
Delegation and A2A: From One Agent to a Production Team
This is where the architecture starts to look different from a normal chatbot.
Delegation lets Hermes hand work to specialized agents. A2A, or agent-to-agent communication, potentially gives those workers a cleaner way to pass structured results between one another. Instead of forcing one AI to carry every detail of the project at once, Hermes can act more like a production director supervising specialists.
Computer Use: The Practical Automation Layer
Computer use may be the most practically powerful capability in the list. Hermes can potentially operate Windows applications with mouse and keyboard input, work through file dialogs, interact with local software and handle interfaces that were never designed for automation.
That gives us an important architectural principle: we do not need to turn every application into an API. If a specialized local program already works well, Hermes may simply operate it or launch it through the terminal.
Spotify, Home Assistant and Other Endpoints
Spotify creates another possible endpoint for the audio side of our media package. Home Assistant is not central to production, but it demonstrates that Hermes can extend beyond content software into physical dashboards and automations. Yuanbao is currently less important to our workflow, but it reinforces the broader idea that Hermes is designed to work across systems rather than remain trapped inside a chat window.
What This Looks Like as a Complete Factory
One URL In
The human-side goal remains almost absurdly simple. We should eventually be able to give Hermes one URL and say, Run the Deep Dive workflow.
The finished project should be capable of producing three videos, two full blog posts, a Spotify episode, YouTube uploads, Facebook posts, TikTok content, thumbnails, captions, metadata, final links and verification.
The human should not have to manually move files through ten different applications. The AI should operate the Factory.
Hermes Is the Director. OpenMontage Is the Editor.
One architectural point became clearer today: Hermes should not replace every specialized application. That would make the system less reliable, not more.
Hermes is the director and orchestrator. OpenMontage is the specialized media assembly and rendering engine. The Factory tracks project state. Hermes decides what should happen next. Specialized tools perform the work they are best at. Then Hermes verifies the result before the workflow moves forward.
Enabled Does Not Mean Running
Turning on these tools does not mean all of them are constantly consuming CPU, memory or GPU resources. Enabled means Hermes is allowed to call them. The real performance issue is concurrency.
CPU-heavy jobs: limited concurrency.
Light research tasks: parallel when useful.
That is how we can give the AI a broad toolbox without letting it hammer the workstation unnecessarily.
The Next Phase Is Verification
Today was about capability. The next phase is reliability. Every newly enabled tool needs to answer three questions: Does it actually work? Does it run locally or require a cloud service? Does it cost money?
Our Factory remains local-first. Whenever possible, we want local models, local applications, local storage and local processing, with cloud services used only when the value justifies the dependency.
From AI Assistant to AI Operator
For years, the standard AI pattern was simple: a human asks a question, the AI gives an answer, and the human does the work.
What we are building is different:
The output is no longer merely text. The output becomes completed work.
The Real Test
The tool list is impressive, but the list itself is not the finish line.
The real test is whether Hermes can take a single URL and reliably turn it into a complete Deep Dive production package without us babysitting every stage.
URL IN → RESEARCH → PRODUCTION → QUALITY CONTROL → PUBLISHING → VERIFICATION → DONE.
If that works reliably, we will have built something much more interesting than an AI content generator.
We will have built an AI-operated media Factory.
Comments
Post a Comment