Published: 7 November 2026
Evidence cutoff: 5 November 2026
Tested tiers: Otter Basic and Fireflies Free
Evidence status: Partially executed
The short answer
For the uploaded English-audio fixture and free tiers we tested, choose Otter if your priority is reliably preserving decisions and producing useful assigned actions and deadlines. Consider Fireflies if its stronger meeting-content model-training restriction outweighs its less consistent action classification, weaker raw-date accuracy, and Free-tier export limits.
Under our bespoke frozen weighted rubric, Otter scored 86/100 and Fireflies scored 71/100. These are comparative benchmark scores, not standardized product-quality ratings. Both tools produced useful meeting notes. Otter won because it preserved buyer-critical transcript facts more consistently and produced the strongest retained downstream result, while also imposing less setup friction.
This was not a feature-table comparison. We uploaded the same synthetic noisy meeting to each product three times, then checked transcripts, summaries, actions, follow-up writing, factual retrieval, exports, privacy documentation, and account friction.
Disclosure
We tested Otter Basic and Fireflies Free using venture-only accounts and synthetic audio. The vendors did not sponsor this comparison, provide payment or benefits, review the article, or influence the scores. We paid no subscription fee for the benchmark. If affiliate links are added later, they will be labelled clearly and will not change the testing method, scores, or recommendation.
Our recommendation
Pick Otter for dependable decisions and actions
Otter preserved the meeting’s essential guidance in all three transcripts. In run 3—the run for which we retained the complete downstream Action Items, follow-up, and retrieval evidence—its Action Items retained the three assigned owners and exact deadlines, its follow-up passed all 9 of 9 checks, and its retrieval answers passed all 5 of 5 questions with timestamped supporting quotations. Equivalent downstream artifacts were not retained for Otter runs 1 and 2, so those results should not be read as three-run action or retrieval repeatability.
The raw transcripts were not flawless. Otter changed “Luis” to “Louis” once and rendered “commissioned deliverable” as “commission deliverable.” Neither defect changed the meeting’s recorded decisions or our buyer guidance. More importantly, the product consistently retained the corrected survey count of 1,408, the November 20 launch date, the $28,000 excluding tax budget, the rejected blue proposal, the charcoal-and-yellow decision, and the fact that video was not approved.
Otter’s main usability caveat is presentation. Its visible overview was selective and could omit owner/deadline detail that was present in the dedicated Action Items section. Buyers should inspect the action surface rather than treating the overview as the complete record.
Consider Fireflies when training restrictions are the deciding factor
Fireflies’ generated Notes were impressive: they preserved all 10 of 10 frozen ground-truth items in every repeat. Its first retrieval run also passed all checks.
The problem was consistency beneath that strong Notes layer. All three raw transcripts omitted the day 20 from both spoken November 20 references, and two overlap-adjacent clauses were repeatedly truncated. The product’s Notes and AskFred feature restored the correct date, but repeatable raw-transcript defects matter when the transcript is the audit trail.
Fireflies also changed which statements it treated as actions. Its three runs produced 5, 7, and 5 to-dos. Every run retained the three fixed owner/deadline assignments. The added items were grounded in the meeting, not fabricated, but the tool inconsistently promoted undated coordination statements into actions. Follow-up scores were 9/9, 8/9, and 8/9, while retrieval scored 5/5, 5/5, and 4/5.
That makes Fireflies useful for readable generated notes, but less dependable when a team needs a stable list of who owes what by when.
Head-to-head results
| What we measured | Otter Basic | Fireflies Free |
|---|---|---|
| Bespoke frozen weighted score | 86/100 | 71/100 |
| Valid repeated uploads | 3 | 3 |
| Critical guidance preserved | Stable across all runs | Stable in Notes; raw-date defect repeated |
| Follow-up checks | 9/9 in retained run 3 evidence | 9/9, 8/9, 8/9 |
| Retrieval checks | 5/5 in retained run 3 evidence | 5/5, 5/5, 4/5 |
| Action-set stability | Run 3 complete; repeat evidence not retained | Fixed assignments stable; 5, 7, then 5 total to-dos |
| Downloaded-file fidelity verified | No | No |
| Direct test cost | $0 | $0 |
| Free retest capacity remaining | None | None |
The weighted score used criteria fixed before execution: word and speaker accuracy; decisions, actions, owners, and deadlines; corrections and dates; summary usefulness; repeatability; setup friction; effective cost; and export, retention, and deletion controls.
Privacy changes the recommendation
The output winner is not automatically the privacy winner.
Otter’s Privacy Policy says it may train proprietary AI on de-identified recordings and transcriptions. We did not verify a Basic-tier opt-out. Otter separately says third-party AI providers do not train on or retain its API customer data, and it publishes a clear named subprocessor list.
Fireflies states that meeting content is not used to train internal or external generative-AI models, while still permitting de-identified or aggregate service data uses. That is a stronger first-party meeting-content model-training restriction than Otter’s disclosed policy, although the two policy formulations are not perfectly symmetrical. Our processed meeting also displayed “Teammates & Anyone with Link” even though we had not taken a public-sharing action. We did not share the link, and all test data was synthetic. A Fireflies buyer should still verify recap privacy before using the tool with sensitive meetings.
Fireflies’ “Zero Data Retention” language also needs a narrow reading: it concerns third-party vendors after processing, not immediate deletion of the meeting from the user’s Fireflies account.
Both vendors put responsibility for participant consent on the customer. We tested uploaded synthetic audio rather than a live meeting bot, so participant-facing bot notices were not verified.
Export and free-tier economics
Neither free workflow proved sustainable for quarterly independent testing.
Setup friction also differed. Fireflies required Google OAuth with read-only Calendar access and offline access for the isolated venture account; that consent requirement contributed to the score gap. Otter’s email-only route avoided calendar synchronization.
Otter exposed a correct TXT export preview, but Browser Relay timed out after export confirmation; download completion and local-file creation were not verified. Its Basic account allowed three lifetime imports, all consumed by the benchmark.
Fireflies exposed multiple export formats, but selecting Markdown triggered “Upgrade to download meeting.” All three free meeting credits were also consumed.
The direct cash cost of this test was $0, but that should not be mistaken for a repeatable zero-cost workflow. Otter exhausted its three lifetime imports, while Fireflies exhausted its three free meeting credits. The approved planning floor for quarterly paid retesting is $34.99 per cycle / $139.96 annually before tax and FX. This budget approval does not itself authorize a purchase.
How we tested
We created one synthetic meeting with four speakers, background noise, overlap, corrections, negation, names, dates, a budget, visual decisions, explicit owners, exact deadlines, and deliberately missing information. The same MP3 with SHA-256 ab4fc24531a1d83f9a9511a96e2475e0c489663059af9e12620394f1b78a3db4 was used for all six product runs.
Each tool received three independent uploads on its free tier. We graded outputs against a frozen answer key and tested:
- transcript and speaker separation;
- summary and action extraction;
- a constrained client follow-up;
- five factual retrieval questions;
- export availability and fidelity; and
- retention, deletion, training, consent, administrative controls, and subprocessors.
We counted one uploaded meeting as one product run. Follow-up, retrieval, and export checks were scored inside those six runs rather than inflated into separate run totals.
Limitations
- We tested uploaded English-language synthetic audio, not live bots or real participant conditions.
- Speaker streams were separated, but neither tool automatically mapped the generic speakers to the four scripted names.
- Local export-file fidelity was not verified for either product.
- Human handling time for the original product runs was not instrumented, so the full mature-dossier production-time gate remains open.
- Otter exhausted its three lifetime imports and Fireflies exhausted its three free meeting credits, preventing a same-account no-cost retest.
- Pricing and policy claims were recaptured from first-party pages on 7 November 2026.
- Product plans, policies, and outputs can change after the November 5, 2026 evidence cutoff.
Bottom line
For a small team choosing between these tested free tiers for uploaded English audio similar to our fixture, Otter is the better provisional default recommendation. It delivered the more stable meeting record and the strongest retained action-and-retrieval result.
Fireflies remains a credible alternative for teams that prioritize its meeting-content model-training restriction and are prepared to verify recap privacy, tolerate less stable action classification, and pay for exports when needed.
Neither tool earns an unconditional recommendation. Otter’s training policy is a material caveat; Fireflies’ raw-date defect, action instability, sharing posture, and export gate are material caveats. Buyers should choose based on which failure mode is less acceptable for their meetings.
Pricing and policy claims were last checked on 7 November 2026. Product details can change; verify current terms before buying.
