AI agents perform poorly when asked to automate tasks drawn from real jobs, according to a benchmark published this month by Marketing AI Institute.

The evaluation, titled "New Benchmark Shows AI Agents Perform Poorly When Automating Real Jobs," is framed as a reality check for buyers being pitched "agentic" automation. The headline finding is blunt: agents that score well on conversational demos and general LLM leaderboards collapse when they must chain tools, read documents, and make judgment calls on real occupational work.

What does the benchmark actually test?

The test set draws from real job descriptions and day-to-day work tasks rather than academic puzzles or synthetic sandboxes. The premise is straightforward: if vendors keep promising that agents will replace human workers, somebody should measure whether the agents can actually do the work.

That framing matters. Academic benchmarks reward narrow competencies. A model can ace a graduate-level exam without knowing how to process a refund, reconcile a ledger, or route a customer escalation. A real-job benchmark collapses the difference.

Who cares about results like these?

Operators do. Procurement teams at mid-market and enterprise buyers — the readers Marketing AI Institute targets — increasingly demand task-level evidence before signing AI contracts. The audience skews toward marketing and growth leaders, in-house agencies, and operations teams already spending on conversational assistants, content tools, and predictive platforms.

A negative result gives those buyers a defensible reason to slow a deal or push a vendor for proof.

How should buyers read a negative benchmark?

Three takeaways:

• A high score on a general LLM leaderboard does not predict job-level competence.
• Vendors quoting "agentic" capability should provide task-specific success rates and named failure modes.
• A short pilot with defined success criteria beats paper benchmarks for procurement decisions.

The third point is the one seasoned operators already know. Benchmarks are signals; pilots are evidence.

What's actually in the public summary?

The available coverage does not list specific model names, raw pass rates, dollar figures, or named enterprises that took part. Readers who want those numbers will need to track the underlying paper or wait for Marketing AI Institute's longer write-up.

That itself is a small operator signal. A benchmark without published scores sits closer to a press release than to a reproducible evaluation.

What should operators watch next?

• Whether the benchmark authors release the full task set so independent teams can reproduce results.
• Whether named vendors respond with their own evaluations, dispute the methodology, or stay silent.
• Whether procurement teams cite the findings in 2025 vendor reviews, or treat the result as one data point among many.
• Whether follow-on benchmarks target specific verticals — sales ops, customer support, finance, software engineering — where agentic claims run hottest.

The bottom line

An independent benchmark says today's agents are not yet ready to replace workers in real job contexts, regardless of how polished the demo looks on stage. Buyers already running pilots will treat this as confirmation. Buyers still evaluating demos will treat it as a reason to ask harder questions before writing a check.

The full evaluation — including task corpus, methodology, and score breakdowns — will decide how seriously the wider operator market weighs the verdict.