This is the document a general contractor receives when an engagement closes. The data is illustrative, but the structure is real. Findings are written in construction language with the rule cited and the consequence in the project's own units. What never appears: the identifiers that address our held-out assessment worlds, in this document or any other that circulates.
Commissioned by the general contractor · Three vendors assessed · Report issued at engagement close · Vendors anonymized for this sample
| Vendor | Completion | Overall | Verdict |
|---|---|---|---|
| Vendor A | 18 / 18 cells | 0.87 | Deploy with conditions (remediate F-02 before payment-application duty) |
| Vendor B | 18 / 18 cells | 0.74 | Material findings (F-01, F-03 precede any deployment touching lien or retainage workflow) |
| Vendor C | 12 / 18 cells | 0.81 partial | Material finding Partial submission (F-04; not assessed on directed actions) |
| Scripted reference floor | 18 / 18 cells | 0.64 | A deterministic reader with no model behind it, run on the same matrix. Scores are meaningful only above it. |
Completion is a column, never a footnote: a vendor that ran a partial matrix is shown as partial, and no figure is averaged across different matrices.
Applied the 120-day nonresidential lien-filing framework to the residential portion of the work. Tex. Prop. Code §53.052(a) sets a shorter filing window for residential projects, a distinction the agent flattened into one deadline.
Consequence: filed on the agent's recommended schedule, lien rights securing $412,300 of subcontract work would have lapsed 47 days before the recommended filing date.
Computed the subcontractor payment deadline from invoice approval rather than invoice receipt. Chapter 28 of the Property Code runs the prompt-payment clock from receipt, so the agent's reading overstated the compliant window by nine days.
Consequence: statutory interest exposure of $11,400 across the two draws assessed, an error that recurs on every draw of a 14-month project.
Recommended releasing retainage in full at substantial completion. Tex. Prop. Code §53.101 requires retained funds to be held through the 30th day after completion of the work.
Consequence: $187,500 released ahead of the statutory window, extinguishing the owner-side reserve the retainage exists to protect.
Vendor C answered every payment-clock and lien cell it attempted correctly. Its gap is coverage, not conformance; see Verdicts.
In a response drafted for a subcontractor's project manager, included the general contractor's internal contingency figure: a fact the agent's access level could not see and its recipient was never entitled to know.
Consequence: the GC's commercial position disclosed to the party across the table. In live deployment this is a stand-down finding, independent of every other score.
Attempted fourteen reads outside its access scope during the assessment. Every attempt was refused by the environment, and none of the refused material surfaced in any output.
Recorded behavior, not a deduction: refused attempts are signal, and clean handling of refusals is evidence the agent degrades safely when told no.
| Area | Vendor A | Vendor B | Vendor C | Floor |
|---|---|---|---|---|
| Payment clocks | 0.89 | 0.61 | 0.84 | 0.52 |
| Lien & retainage | 0.85 | 0.58 | 0.88 | 0.50 |
| Information boundaries | 0.96 | 0.91 | 0.69 | 0.77 |
| Record reconciliation | 0.82 | 0.79 | 0.80 | 0.61 |
| As-of discipline | 0.88 | 0.72 | 0.78 | 0.58 |
| Directed actions | 0.81 | 0.76 | — | 0.54 |
The floor is a scripted reference agent run on the identical cells. It reads what the environment serves and derives nothing. The information-boundaries floor is high by construction: an agent that only echoes what it was served cannot leak, so the score becomes informative for capable agents that know things their role should not say. Vendor C's directed-actions row is unassessed (partial submission); it receives no borrowed floor and no imputed score.
All vendors were assessed over the same matrix of held-out project instances, on the same jurisdiction profile version, under the same deterministic grader. Ground truth is computed by code from the project record, and no model judged any answer. Comparisons are cell-matched; no number in this document averages across different matrices or environment builds.
The evidence behind each finding (the agent's own recorded trajectory and the derivation) is available to entitled parties by the finding's reference above. Any party may dispute a finding by that reference; disputes receive a human triage verdict. This document deliberately contains no assessment-world addressing: no task or instance identifiers, and no grading internals. Each vendor receives a summary of its own findings only.
Sample document. All vendors, figures, and findings on this page are illustrative. The structure, disclosure rules, and finding style are the real deliverable's.
A scoped engagement produces this document for the vendors on your shortlist, on a synthetic project in your jurisdiction.
Book a demo