so apparently they automatically hand over your email to Slack to contact you upon login even tho I don't remember this being mentioned anywhere at all. I went to check the Momentic settings and theres a Notifications section with Slack turned off...
Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).
You're right stock CC was pretty terrible. Thats why we compared against CC && Codex with agent browser and playwright mcp as well as devin. Do you have any other agents / configurations you'd like to see?
We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench
I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.
This sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.
Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.
- I would always pick an open source tool I can strap to my custom harness over using a service like this. Do you somehow expect that no such competitor will arrive that performs as well?
- If you are YC 2024 what have you been doing for 2 years?
- The real edge here would be supporting mobile development builds on real devices, is this on your roadmap and if so what's your plan to support it? In that case I would pay for this if you handled the real device on your end.
Hey! I'm Henry, one of the engineers working on Mo. To answer your questions:
- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.
- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.
- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.
> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.
Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
I feel like your writeup could be a little stronger if you made it clear how important is hitting those recall targets.
I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.
Congratulations to the team tho! Those are very nice results
Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.
I tried to use mo on the platform (I have been wanting to use momentic for a while). There was no onboarding really, now it's trapped in me needing to configure auth but I don't have time to click through 20 pages to find out how to do that :( I really wanted to like you!
Hey there! Sorry to hear that the onboarding experience didn't meet your expectations. Would you mind reaching out to support@momentic.ai and sharing your account details so that we can debug this? We'd love to help you get set up and start using Momentic.
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
Heyo! I'm Jeff, one of the co-founders here. Mira itself is for code review and does no e2e analysis. We actually automate your live web app or mobile app, so we're catching a whole class of issues that are just not possible with static analysis (is the button cut off in the UI? does your AI agent respond correctly? does the video render correctly? etc.)
But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).
The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.
Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
When people say "just point Opus/Codex at localhost using Playwright MCP," they've obviously never attempted to do exploratory testing by leveraging parallel browser workers against stateful applications. A coding agent running in a loop spends almost all of its context window struggling with DOM selectors and doesn't even detect state persistence problems like the 2.5h->25h decimal inflation upon refreshing the page as demonstrated in the demo.
Managing read/write resource leases for parallel sub-agents not to step on each other's test fixtures is already a non-trivial orchestration challenge by itself.
Agreed! We've also found that the tooling you use makes a big difference in how efficient and precise the agent is, which is why we included the comparison of Playwright MCP vs agent-browser. When we were first starting Mo we tried using several different browser interfaces but we weren't able to get the recall we wanted. We settled on rewriting some custom tooling based on our mcp for our investigatory/exploratory agents. Tooling about how to interact with the A11y and DOM, breaking out easy CUA subproblems, and browser infra are still all non-trivial problems.
The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.
Hey, Jeff I'm one of the co-founders. Totally agree for a small side project, having a single sub-agent that focuses on vibe testing is enough.
But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.
As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.
Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.
So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.
It would be really nice to be able to plug in the subs from the labs. The industry seems to be moving that way even if its still super limited. Ex https://loginwith.ai
That is great. But hard to implement for side projects or independent devs, its a partner program that they can only approve the businesses they deem acceptable.
I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.
tried testing it with my ios app but got:
"The selected XXX build (com.XX.XX) cannot be installed because it is incompatible with the hosted iOS simulator. No XXX behavior could be tested."
would be great to have more details about whats not compatible, I assume iOS version? app requires 27
Hey! I'm one of the cofounders. Unfortunately I think you're right that this is due to our simulators being on iOS 26. We're actively working on supporting 27. Let me see if we can get that out now. I'd love to help you get unblocked; have you set up a Slack connect channel with us yet?
Thats awesome, there's tons of cool work being done in exploratory testing right now. We'd love to see how it stacks up against Codex/CC/Mo. If you want to be added to the bench feel free to send over the results/how to repro w/ a pr on qa-bench and we can merge it!
Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.
The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.
Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.
Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.
One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.
Hey! I'm Henry, one of the engineers working on Mo. Currently, Mo is super focused on being the absolute best at finding bugs, but isn't meant to be a general purpose coding agent. For that we ship a CLI (https://github.com/momentic-ai/qa) and matching skill which you can use to integrate Mo with whatever coding agent you prefer. One pretty successful setup we've seen a lot of customers use is telling the agent to build a feature, then use Mo to test it, and only go back to the user for input once all the bugs are resolved.
so apparently they automatically hand over your email to Slack to contact you upon login even tho I don't remember this being mentioned anywhere at all. I went to check the Momentic settings and theres a Notifications section with Slack turned off...
used my public email so idc but others may
We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench
- I would always pick an open source tool I can strap to my custom harness over using a service like this. Do you somehow expect that no such competitor will arrive that performs as well?
- If you are YC 2024 what have you been doing for 2 years?
- The real edge here would be supporting mobile development builds on real devices, is this on your roadmap and if so what's your plan to support it? In that case I would pay for this if you handled the real device on your end.
- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.
- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.
- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.
is this good? so Mo costs more money per bug? I'm confused.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
I also do believe search quality is recall dominated, but the link between the metrics and results are not as clear as they could have been. I think you have a stronger sell here than it looks at first glance.
Congratulations to the team tho! Those are very nice results
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
1. Mo generates its own test cases in natural language, but you can ask it to write Playwright and maintain it in your repository if you really want. We actually do have a regression testing product as well where tests are just natural language in your repository. For that product yes there is some vendor lock-in since this is a proprietary format that only our CLI translates into browser actions. But ultimately since it's natural language it's pretty portable :)
2. Mo is focused on exploratory testing and so is inherently non-deterministic. But the regression testing product has a ton of caching built into it that we've written about in the past (not just a single css selector). https://momentic.ai/blog/teaching-browser-agents-user-intent This is a big part of reliability and latency optimization, otherwise your CI would just hang for hours on any reasonable suite.
3. I would say the regression testing product was very much SDET/developer focused since it required you to have a codebase and use Git. Mo is meant to be super easy for anyone to onboard onto and lower the barrier to entry, so we're hoping we'll see more product owners, PMs, QA and even leadership use the product. But devs are always going to be our bread and butter, we ultimately tried to build the product we wanted to use ourselves.
4. We're optimized for recall (# of bugs you find in a given amount of time). The bench shows we find ~3-5X more bugs compared to Codex and Claude Code, with approx. the same precision. The service costs more than the raw harnesses for sure, but that's the all-in cost including our infra, parallel agents, etc.
What do you guys have that's better than a self-hosted, open source extensible QA AI tool like Mira?
[1] https://miracode.ai
But to your general point, this benchmark is showing we are finding 3-5X more bugs than the best OSS performance out there right now. We did also try it on tester army e2e and it underperformed Codex + Playwright MCP (this will make it in the benchmark soon).
The "better" comes from custom agents and massively parallelized infrastructure. I'm not sure how many folks want to or even can self-host fleets of GPUs, browsers and Mac minis. We're happy for folks who can but our current offering is for those who want to find the most bugs with the least amount of effort.
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
This strikes me as a company that is late delivering a product for November 2025, when it is November 2026.
But I would say even for medium sized features, it's the same difference between asking your Codex to "review the code make no mistakes" and using a paid AI code review tool.
As your product surface area increases, you need fundamentally different agent architecture to maintain context over time and across bug bash sessions. The benchmark apps are probably among the simplest apps we test on, and even those show a massive perf. gain. For large companies with 1000s of important user flows, have to support different viewports, and need to work across mobile/web/desktop, asking Codex to test it all before a big launch is not scalable.
Even basic constraints like the # of browsers/devices you can run concurrently, the amount of reliable TPM you need, and how many agents can run on a single box become problematic. You also have to think about ktlo: how do you evaluate new model releases in a data driven way, how does agent become more efficient itself over time, etc.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
would be great to have more details about whats not compatible, I assume iOS version? app requires 27
Everything has to be reproducible and runnable by the general public
One suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
how does this fit in to fixing the bugs? how do we close the loop with the agents we have coding?