An AI that cannot hallucinate read 135 tender positions. We scored it 7.5 out of 10.

A bill of quantities for a school has 1,500 positions. A window maker prices 60 of them. Before anyone can interpret, match or price a single position, someone has to answer the dull question 1,500 times: is this one ours?

In DatumOS that answer is a field on every position, scope_status: in scope, out of scope, or uncertain. Until last week nothing set it. Every position defaulted to "in scope" and the estimator sorted it out by eye. This post is about the model we tested to fill that field, what it got right, where it hesitated, and why we are going to use it anyway.

Everything below is synthetic. The positions were written by us to look like the ones clients send, and no pilot company's tender appears here.

A model that answers with a number, not a sentence

The usual way to classify text with AI is to ask a large language model and parse whatever it writes back. It works, and we do it elsewhere in the product, but it has two costs that matter at 1,500 positions a tender: it is slow and expensive per call, and the answer is a string. A string can be anything, including a confident wrong one.

TypeSafe AI released a different kind of model in September, which they call System One. Their first public one is named Jev. It does not generate text at all. You send it a structured state, a JSON object, plus one or more typed questions: pick one of these options, score this on a scale, or yes/no. It returns a typed answer for each, with a probability that is trained to be honest. When it says 0.95, it should be right about 95 times in 100. When it says 0.5, it is telling you it does not know.

That last property is the interesting one for a scope filter. An estimator does not want a classifier that guesses. They want one that says these 40 are yours, these 1,400 are not, and these 60 you should look at.

The call looks like this, per position:

{
  "state": {
    "account_trades": "Window and door contractor (PVC, timber, aluminium): windows, entrance doors, interior doors, frames, sills, roller shutters ...",
    "document": { "trade_package": "Fenster, Außentüren und Verglasungen" },
    "position": {
      "oz": "06.02.0040",
      "short_text": "Fensterlaibungen innen verputzen",
      "long_text": "Fensterlaibungen innen mit Gipsputz verputzen, ... Putzabschlussprofil am Blendrahmen ...",
      "unit": "m", "quantity": "118",
      "toc_path": "06 Putzarbeiten"
    }
  },
  "questions": {
    "in_scope": { "type": "noul", "instructions": "Would the contractor described in account_trades be expected to price this position as part of their own trade?" },
    "trade":    { "type": "choice", "criteria": { "fenster": "...", "tueren": "...", "mauerwerk": "...", "daemmung": "...", "beton": "...", "sonstiges": "..." } }
  }
}

Two questions in one call. Is it ours? depends on who is asking, which is why the contractor's own description of their trade is part of the input; a door-only company and a window-and-door company get different answers for the same position. Which trade is it? does not depend on who is asking. The example above, plastering the window reveals, contains the word "Fenster" three times and belongs to the plasterer. The model returned 0.11 and sonstiges (other). A keyword filter would have put it in the window maker's pile.

Each call took about 850 milliseconds and roughly 1,000 input tokens, which at the published price is four thousandths of a cent per position. A 1,500-position tender costs six cents to classify.

Round one: 52 positions we wrote ourselves

We started with positions in our own shorthand, the way an estimator would jot them: obvious ones, keyword traps, abbreviation-heavy ones, and eight that are genuinely ambiguous for a window maker, such as a T30 steel fire door or a post-and-beam curtain wall.

Every trap passed. Fenstersturz Stahlbeton-Fertigteil (a precast concrete lintel) went to concrete at 0.06. Fensteröffnung zumauern (bricking up a window opening) went to masonry at 0.10. Bauendreinigung einschl. Fensterputz (final cleaning including window washing) went out at 0.07. The estimator shorthand KS-Fe. 1-flg. DK ws 1.010 x 1.210 RC2 came back a window at 0.97, as did the same position with the characters mangled the way a bad scan mangles them.

The one miss was a steel door frame, a Stahl-Umfassungszarge for the company's own interior doors, at 0.26. The model knew what it was, the trade answer was doors at 1.00, and still did not want to say the window-and-door company would price it. Keep that one in mind.

Round two: 83 positions the way clients actually write them

Round one proved the model reads German construction text. It did not prove it reads tenders, which are written in house styles that have nothing to do with clarity. So we took the five sample documents from our pilot walkthrough and the GAEB samples in our repository, and wrote 83 positions in their voices:

• Inherited positions. A base position with the full specification, then eight variants that say only "Gemäß Position 2.510." and list the deltas: a different handle, a roller shutter, a taller frame. Some variants do not contain the word "Fenster" at all.
• GAEB short texts. Eight words and no long text: Türzarge Stahl 875 x 2125, BA 03.
• English bills of quantities, dense one-paragraph items ending in a room name.
• Narrative specifications, prose plus bullets plus QUANTITY: 204 No., including whole systems such as an automatic entrance door.
• Contractor schedules with product codes and a sundries section.
• A masonry lot from a multi-trade tender, written in VOB long-text style with Fabrikat: ... oder gleichwertig and Angebotenes Fabrikat: ........ (Bieterangabe) lines, full of window-shaped items that all belong to the bricklayer.
• A formal window lot with wie vor, jedoch references, surcharge and deduction lines, an Eventualposition, a Nur-Einheitspreis rate, and notes about work provided bauseits, by others.

Harder input, a modest drop, calibration intact. Every position from both rounds is on one chart:

The shape is the finding. The clear positions sit in their bands, the ambiguous ones spread across the middle, and no position was put past a threshold on the wrong side. Three results from this round changed how we will build the feature.

Inherited positions need no resolving. We had assumed the pipeline would have to look up the base position and paste its text in before the classifier could work. We tested the same delta-only variant twice, once bare and once with the base text attached. Bare: 0.95. Resolved: 0.96. The section path and the trade header carry it. All eleven inherited cases were correct.

"By others" is read literally. A note inside the window section about the electrical connection for the roller-shutter motors, marked bauseits durch das Elektrogewerk, kein Preis, came back 0.05. A scaffold note, wird bauseits gestellt, keine gesonderte Vergütung, 0.13. In the English contractor schedule, scaffolding "provided by main contractor, NOT in this package", 0.09. In the masonry lot, every window-shaped item went to the bricklayer where it belongs: the window stop in the wall, the lintel, the plaster stop bead at the frame, the XPS in the reveal.

Where it hesitates is a single, nameable pattern. Every direction error in round two, and most of the positions that fell from "in scope" into "uncertain", have the same shape: an item that belongs to the window maker, described with another trade's verb.

The model anchors on the foreign noun, Elektro, Gerüst, Prüfung, and discounts "in the frame of position 04.02.0010" or "the contractor shall provide". It does not flip these to "not ours". It parks them at 0.4 to 0.8, which under our thresholds is the review queue. That is the right failure for a filter to have. It is also more review than an estimator would want.

The lever we had, and how far it moved

The contractor's trade description is part of every call, so we tried the obvious thing: extend it. We added frames, insect screens, "electrical hardware built into our own elements", and "ancillary work on our own elements: measurement, shop drawings, samples, testing, reglazing, scaffolding where the tender requires it from the window contractor". Then we re-ran the seven hesitant positions and, more importantly, four controls: the bauseits electrical note, the electrical point "by electrical contractor", and both scaffold-by-others notes, which the new wording could have dragged upward.

Six of seven rose, by 0.19 on average, all in the right direction. Insect screens went from 0.62 to 0.95. The intercom cut-out went from 0.48 to 0.81. The four controls stayed where they were, between 0.06 and 0.20. So the description is a real lever and it does not leak.

But only one of the seven crossed the 0.85 line. The steel frame reached 0.68 with the word Zargen now literally in the contractor's description. There is a ceiling on how far context can push a position whose verb belongs to someone else, and no amount of adjectives in the description will lift it. That is a property of the model, and we stopped tuning there.

The score card

7.5 out of 10 for the job we would give it. What keeps it from eight: about 30% of positions would land in the review queue, and a third of those are ones an estimator would price without a second look. What keeps it from dropping below seven: in 135 positions it never once put a wrong answer past the confidence line. For a filter that sits in front of pricing, that is the property that matters.

For comparison: the product's current behaviour, everything in scope, scores zero. A keyword filter would score perhaps three on this set, because the traps are the set. A large language model asked the same question per position would likely match on accuracy, without the calibrated probability, at roughly ten times the cost and latency.

What we will build

The scope filter goes in with this model behind it, and three design decisions that come straight from the numbers.

Thresholds at 0.20 and 0.85, and a review queue in between. We tried moving them. Lowering the upper line to 0.75 rescues the reglazing and the window boards, and pulls the fire door and the automatic entrance door into "in scope" with them, which is worse. The queue is the product, not a bug in it.

The trade description is a setting the estimator owns. Two door companies differ: one fits T30 fire doors, one does not. The description is where that lives, it is plain language, and the experiment shows the model respects it without over-reading it. We will ship a default per trade, with the "ancillary work on your own elements" clause included, and the queue will show which wording would have moved a position.

The model classifies. It does not extract, write or price. Positions are still read by a language model, the dossier is still written by one with citations, prices still come from the company's own rules. A model that answers with a number is the right tool for the one question that has a short answer, and the wrong tool for everything else on the page.

Two gates before it touches a real tender. First, a batch of positions from an actual pilot tender with the estimator's own scope decisions as the ground truth; 135 synthetic positions written by the people who planted the traps is a start, not a validation. Second, an answer from the vendor on where the text is processed. Until we have that in writing, tender text stays in the EU, and this stays an experiment.

Who you are dealing with

Four people, all of whom write the code, and who run experiments like this one before anything goes into the product. The case set and the evaluator are a few hundred lines of Python; if you want to run them against your own trade description, ask.

If you price GAEB tenders and want to see which of your positions a model would hand back to you for a second look, request a pilot. We reply within two working days.