Study protocol: study mode, blinding, grading and statistics

How professionals compare answers of any origin – Reporter, plain prompting, other AI, humans – blinded; the check of every single answer in use is described in the process dossier

Version 4.0 · 6 October 2026

1The yardstick – see the process dossier

What is measured is not whether the AI finds the one right solution, but whether it asserts something false – checked against the official sources themselves. How the program checks every answer in practical use – citations against the entries, official look-up in RIS, Findok, NeuRIS and Cellar, verification report with a status per citation –, and how the request is prepared and made beforehand, is described in process dossier: preparation, query and validation. This dossier is devoted solely to the study protocol: the blinded comparison of answers of different origin, grading, statistics and documentation. Expert documents as a gold standard are an optional scientific add-on (chapter 4).

2A platform, not a study

IUCOM Reporter sees itself as a classic analysis tool. It does not deliver “the one” study but the tool with which university chairs, law firms, legal departments and administrations set up their own methodologically sound comparisons. The Reporter's own pre-processing is one of several conditions that can be tested – not the result to be proven.

Outcome-based rather than system-based. What is tested is what a technology actually delivers in use – the finished answer –, not how it is built internally. Hence any technology can be tested, including one whose inner workings nobody discloses: import its answers and measure them blinded – against the sources and, as an add-on, against a gold standard.

A proven model from medicine. This is exactly how diagnostic accuracy studies are set up there: a test is examined as it is used in practice and compared with a reference standard – how it works internally does not matter. For reporting there is an internationally recognised guideline, STARD 2015. With the gold-standard add-on the study mode transfers this principle to law: the technology tested is the index test, the expert document the reference standard.

The method is recognised. With GDPval (2025) OpenAI applied exactly this principle: experienced professionals compare AI results blinded with work produced by humans, without knowing which is which – 1,320 tasks from 44 occupations, including legal briefs. It was a one-off study, not a tool for others. For the study mode this is an argument, not competition: the grading approach matches the state of research – the study mode makes it repeatable for every professional.

The unique selling point lies in the combination. Individual building blocks exist elsewhere: general annotation tools in which humans grade imported answers; evaluation platforms for developers that compare AI outputs against reference answers; providers of legal benchmarks. Rarely found together: a tool for lawyers instead of programmers, with built-in blinding, randomisation and agreement measures, that makes human and machine answers equally comparable, considers the data protection of participants and brings its own rule-based pre-processing as a testable condition – for the German-speaking legal area.

3The pre-processing: one condition among several

At its core the pre-processing is a rule-based retrieval stage: assisted by AI it searches secure sources for the request – official databases, statutes in the version in force, legislative materials, literature, own documents –, combines request and sources with the rule set into a synthesis and sends it to the language model as an enriched request. The rule not to invent anything is part of the rule set. It is an instruction, not a guarantee – which is exactly why the test measures how often the goal is achieved.

4The gold standard – optional scientific add-on

Only for the test phase. In practical use the checking rules apply (chapter 1); the gold standard is an add-on of the study mode for questions of completeness and professional quality. The study mode runs without it as well – then answers are checked against the sources. Category and degree of agreement exist only with a gold standard, because otherwise they lack a yardstick; without one the program does not ask for them at all. The overview:

The gold standard consists of one or more expert documents – opinions, statements, model solutions by qualified persons. They are kept per area of law in the folder goldstandard next to the own documents of that area (program folder Kategorien\<area of law>\goldstandard; the button Gold-standard folder opens it). Readable formats are .txt, .md, .docx, .doc, .rtf and .odt. Optionally the gold standard can be attached to the Reporter runs – only as a working aid: the AI then sees the model solution, and the comparison measures how well it adopts it, not its own performance. Results with the attachment do not count as a measurement; the report marks them as such.

How many documents the yardstick rests on. Experts often disagree among themselves; a single expert document is a single professional opinion. The report therefore states how many expert documents the gold standard rests on and warns if there is only one. Several independent documents as a consensus are preferable.

5The conditions

• Reporter with pre-processing – 1 to 10 runs, every answer goes into the archive.
• Control: model without pre-processing – the same model, asked directly but with the instruction not to invent anything. Comparing it with the Reporter isolates the contribution of source search and rule set.
• Imported finished answers – by experts, students, trainees or other AI providers. Accepted are Word (.docx, .doc), PDF, RTF, ODT, text, CSV and Excel tables (one answer per row) and JSON exports from APIs (OpenAI, Anthropic, Mistral, Google and own fields answer or text). Scanned handwriting contains no text – the program says so; such answers have to be typed up.

Origin details. On import, per answer – for AI: provider, model, version, date and the prompt used; for humans: qualification level (student, trainee, experienced professional, other professional), whether aids were allowed, and the consent. Tables and JSON exports can bring these details per row (columns answer, source, provider, model, version, date, prompt, aids, id, consent). The details stay hidden during grading.

Other AI results with their own question. Instead of an archive case, an own question can be entered – for instance the question that produced the results of other AI providers. If an imported answer brings its prompt (table, JSON export, origin window), the program takes it over as the question. This question then also goes to the Reporter and the control, so all answers answer the same question.

Clean assignment. Every answer belongs to exactly one case – the archive entry chosen or the own question. The exact wording of the question the answer was produced with is recorded. If it differs from the facts of the case, the program warns on import and the report states it per answer – whoever was asked differently is only comparable with reservations.

6Blinding

As soon as more than one condition is involved, everything is blinded – for the AI grading as for human reviewers:

1. Normalise. Imported answers often reveal their origin through outward features: AI answers through markdown and typical phrases (“Certainly!”, “I hope this helps”, disclaimers), human ones through letterhead, place and date, greetings, signature. The program removes these features by rule and brings all answers into the same plain text format – without touching the content.
2. Align on request. Typing errors and style cannot be neutralised by rules. Optionally the AI aligns spelling and style – with the strict instruction to keep content, structure, sources, figures and amounts verbatim, even if wrong. If it visibly shortens an answer, the alignment is discarded. Afterwards the program checks in code whether the same file numbers and sources, the same provisions, the same figures and amounts (whatever the notation, e.g. 12.500 or 12500) and the same placeholders appear before and after the alignment. If anything is missing or added, the alignment is discarded and the answer stays verbatim; the log names what would have changed.
3. Pseudonymise. Personal details are replaced by placeholders – always when human answers are involved, and then equally for all answers, so that the placeholders themselves do not reveal the origin.
4. Random order and codes. The answers are put in random order and receive blind codes X, Y, Z … (never A and B – they stand for the two grading systems; after unblinding each code appears in the report in the colour of its origin: green Reporter with pre-processing, orange model without pre-processing, violet other AI, blue professional, turquoise students and trainees – with a hint on hover, in HTML and Word); the random seed is logged, so the order is traceable.
5. Grade blind, then unblind. The AI sees only the facts, the coded answers and – if added – the gold standard. Only after grading are the origin details switched on: the report assigns every code to its condition and summarises the results per condition.

7Grading: two stages, with hallucination rates

First, answers are measured against the sources – this always applies:

• Two hallucination rates per answer, stated expressly, even if very low: the source rate (demonstrably invented sources and miscitations, relative to all cited sources) and the fact rate (invented facts or figures, relative to all checked factual and numerical statements). Separate, because sources and facts have different denominators – added up, the rate could be distorted or exceed 100%. “Not verifiable” is not “invented”. Both rates are determined by the grading AI in the study mode by comparison with the facts and the sources – not by the program code. Checked in code during use are the cited decisions (chapter 1); invented facts are not recognised by the code.
• Two grading systems side by side: legal opinion catalogue by analogy with the standard of care of §§ 1299, 1300 ABGB (bars K1 to K9 before quality Q1 to Q8) – these provisions govern the liability of experts, not quality criteria; their standard is transferred, just as STARD is transferred from medicine and university grading grid (Assurance of Learning).
• Separate evaluation of the retrieval stage for the Reporter runs: stage 1 – are the delivered sources relevant, and with a gold standard: does the pre-processing find its key sources (hit rate)? Stage 2 – does the model render the delivered sources faithfully? Plus the cause of every departure: retrieval or processing. This check runs as a request of its own, so that the source lists do not betray the blinding.
• Reasons and proposed solutions: for every departure the analysis gives the reason – why the statement is wrong, incomplete or unsupported, with the decisive passage from the provision or case law, or with a gold standard from it – and a concrete proposed solution: the correct statement, the missing point or the correct source. At the end it says what should change in the question, sources or procedure.
• Blinding check of the AI: at the end the grading AI guesses “human or AI” per answer; the program compares the hit rate with chance.

Only with a gold standard (optional, chapter 4) are added:

• Comparison with the gold standard per answer: result matching, missing key points, contradictions, degree of agreement in percent.
• Categories as in the Stanford study (Magesh et al. 2025): correct, incomplete, hallucinated – so that the figures are comparable with the most important reference.

The AI as a grader – pre-sorting, not verdict. Language models are known to be biased as graders: they favour answers from their own model family [19; 23]. If, say, Claude grades answers among which there are Claude answers from the Reporter, this objection is obvious. The program therefore keeps three rules:

• A different model family. If the configured model belongs to the same family as AI answers being tested and the key of another provider is available, that provider's model grades automatically. Otherwise the report warns expressly.
• A measure of its own. ‘Import ratings’ compares the blinded AI verdicts with those of the humans: hit rate and Cohen's kappa per reviewer, Krippendorff's alpha without and with the AI as a further reviewer. So ‘the AI grades’ becomes a supported ‘the AI grading agrees this much with the experts’.
• Ranking. The AI grading is a pre-sorting; the verdict lies with the human reviewers of the blind folder – people have the final say.

8Human reviewers: blind folder and agreement

An AI grading creates order – humans decide. For this the program creates a blind folder in the study folder (Documents\IUCOM Studien): the question, the gold standard (if added), the normalised answers under their codes, a rating sheet (CSV, opens in Excel) and instructions. Per code reviewers enter: with a gold standard the category and the agreement with it in percent, always the grade from 1 to 5, the grid level (1 to 4 or blocked, as in system B) and – only after grading – the guessed origin (human or AI). The key code → origin is kept separately in the folder Key – study lead only.

The button Import ratings evaluates the filled sheets (CSV or Excel, any number of reviewers):

• Agreement between reviewers: Krippendorff's alpha for the categories, the degree of agreement, the grade and the grid level (ordinal) – it copes with any number of reviewers and missing values; with two reviewers also Cohen's kappa. Reading per Krippendorff: from 0.80 reliable, from 0.667 tentatively acceptable.
• Blinding check: how often were the reviewers right with their origin guess – compared with what mere guessing would yield at these proportions? If they are not above chance, the blinding is proven.
• Unblinding: the results per condition – shares correct, incomplete, hallucinated, mean agreement and grade – and per answer the verdicts of all reviewers.

9Report following STARD – by analogy in law

Diagnostic accuracy studies examine a test as it is used against a reference standard – reported following the STARD 2015 guideline. In law the study mode works by analogy with STARD: the index test is the answer of any origin, the reference standard is the gold standard. The Reporter creates this report itself:

• Target question: in the "STARD" step a yes/no question is entered, such as "Is the termination effective?". The AI reads the reference from the gold standard, or the study lead sets it.
• Index result: every answer is read as "yes", "no" or "indeterminate" – by the reviewers of the blind folder (majority; the rating sheet then has a column of its own), otherwise from the AI grading.
• Report: "Create STARD report" combines all tests with a target question in a folder – every test is a case. Per condition: 2x2 table, sensitivity, specificity, positive and negative predictive value and accuracy with 95% confidence intervals after Wilson, plus the flow diagram with all exclusions and the 30 items of the STARD checklist: what the program can document is filled in; design, sample, sample size, registration and funding remain marked as tasks for the study lead. Indeterminate results are listed separately and do not enter the table.

10Statistical methods

This chapter describes exactly the procedures the program computes, and only these. The figures per condition are descriptive, some with a 95% confidence interval. The program computes a significance test only for the blinding check (10.5); differences between conditions are not tested for significance (10.8). Numbers in square brackets refer to the references in 10.9.

10.1 Random order and blind codes

From two conditions on, the answers are put in random order with the Fisher-Yates procedure in Durstenfeld's version [8; 13]. The random number generator is .NET Framework's System.Random, a subtractive generator after Knuth [13]. The seed comes from the system clock and is recorded in the study log, so the order can be recomputed. In this order the answers receive the codes X, Y, Z, W, Q, K …; A and B are excluded because they stand for the two grading systems. Only the study key holds the mapping code → origin.

10.2 Outcome measures per answer

The grading AI returns machine-readable values per answer: HERKUNFT and the grid level, with a gold standard also KATEGORIE and UEBEREINSTIMMUNG, with a target question ZIEL. From these the program computes:

• Category (nominal, only with a gold standard), after Magesh et al. [16]: correct (substantively right and supported), incomplete (refusal or essential gap), hallucinated (substantively wrong or wrongly grounded).
• Degree of agreement (percent, only with a gold standard): the share of the gold standard's key points that the answer contains correctly.
• Two hallucination rates (percent), separate because numerator and denominator must match:

HR_sources = (E + F) / N_S × 100    HR_facts = T / N_T × 100

Here E is the number of demonstrably invented sources (bar K2), F the number of miscitations (K3) and N_S the number of all cited sources; T is the number of invented facts or figures and N_T the number of all checked factual and numerical statements. Both rates thus lie between 0 and 100%. “Not verifiable” does not count as invented.

• System A, legal opinion catalogue (by analogy with the standard of care of §§ 1299, 1300 ABGB – liability provisions, not quality criteria): a failed bar K1, K4 or K9 gives level 5. A failed bar K2, K3 or K5 to K8 caps the result at level 4. Otherwise the weighted mean of the quality criteria Q1 to Q8 applies (weights 15, 15, 15, 20, 15, 10, 5, 5%), rounded. An average never offsets a bar.
• System B, university grid (Assurance of Learning [22], applied by analogy): first the integrity of the sources is checked. If a source is demonstrably invented, the answer is blocked. Otherwise each sub-competence B1 to B7 receives a level from 1 (exceeds expectations) to 4 (does not meet expectations). The AI sets the overall level by weighing them, because the grid prescribes no arithmetic rule. It appears at the end of every grading as its own line and is reported in the unblinding per answer and per condition.

10.3 Summary per condition

Per condition the program reports:

• with a gold standard, the categories as absolute frequencies;
• with a gold standard, the degree of agreement as an arithmetic mean;
• the grid overall levels as frequencies per level and as a median. The scale is ordinal, so no mean is computed. “Blocked” is ranked as the worst level.

For the human sheets the program reports the shares per category across all ratings, the means of agreement and grade, and the grid levels as frequencies per level and median across all ratings.

10.4 Agreement between reviewers

Krippendorff's alpha [10; 14; 15] for any number of reviewers, with missing values:

α = 1 − D_o / D_e = 1 − (n − 1) · Σ_u [ Σ_i≠j δ(c_ui, c_uj) / (m_u − 1) ] / Σ_i≠j δ(c_i, c_j)

Where:

• u runs over the answers that have at least two ratings;
• m_u is the number of values for answer u, n the number of all pairable values;
• the sum in the denominator runs over all pairs of all pairable values.

The difference function δ depends on the scale:

• for the category, nominal: δ = 0 if equal, otherwise 1;
• for degree of agreement and grade (1 to 5) on an interval scale: δ = (c − k)²;
• for the grid level, ordinal [14; 15], with “blocked” ranked as the worst level 5. Here n_g is the frequency of level g among all pairable values:

δ(c, k) = ( Σ_g=c…k n_g − (n_c + n_k) / 2 )²

The implementation is checked against Krippendorff’s worked example (four reviewers, twelve units, missing values) and reproduces its values exactly: nominal 0.743, ordinal 0.815, interval 0.849 [14].

Reading after Krippendorff [15]: from 0.800 reliable, 0.667 to 0.800 only for tentative conclusions, below that insufficient.

With exactly two reviewers the program also computes the unweighted Cohen's kappa [5], over the answers rated by both:

κ = (p_o − p_e) / (1 − p_e),    p_e = Σ_k p_1k · p_2k

AI grading against humans. If the blinded AI grading is in the study folder, the evaluation also computes: the share of identical category verdicts of AI and humans, Cohen's kappa between the AI and every reviewer (and their mean), and Krippendorff's alpha for category, degree of agreement and grid level once with the humans only and once with the AI as a further reviewer. If alpha drops with the AI, it departs from the humans.

10.5 Blinding check

After each grading the reviewer guesses whether the answer comes from a human or an AI. The grading AI gives the same guess in the line HERKUNFT. The program counts the correct guesses h among n. The chance expectation follows from the marginal shares – like chance agreement in kappa:

p₀ = Σ_{k ∈ {human, AI}} (n_k^true / n) · (n_k^guessed / n)

The test is one-sided, using the exact binomial test [7] – without approximation, hence reliable with few guesses too:

p = P(X ≥ h) = Σ_k=h…n C(n, k) · p₀^k · (1 − p₀)^n−k

The sum is computed via logarithms and is therefore stable for large n too; checked against table values (e.g. P(X ≥ 6) = 0.1445 for n = 8, p₀ = 0.5).

• Null hypothesis: the hit rate equals guessing.
• If p < 0.05, the blinding is considered not held.
• Below eight guesses the program draws no conclusion.

The procedure follows the idea of the blinding indices of James et al. [12] and Bang et al. [1], but does not compute these indices.

10.6 Diagnostic accuracy after STARD

With a yes/no target question every condition becomes an index test, and the gold standard (or the study lead's determination) the reference [2; 6]. Per condition the program builds a 2×2 table across all cases:

• TP = true positive, FP = false positive, FN = false negative, TN = true negative;
• “indeterminate” answers are not in the table but counted separately;
• cases without a reference are excluded.

sensitivity = TP/(TP+FN)  ·  specificity = TN/(TN+FP)  ·  PPV = TP/(TP+FP)  ·  NPV = TN/(TN+FN)  ·  accuracy = (TP+TN)/N

Every proportion p̂ = k/n receives the 95% Wilson score interval [21], which stays reliable for small n and proportions near 0 or 1 [3; 18]. z is 1.959964:

[ p̂ + z²/(2n) ± z·√( p̂(1−p̂)/n + z²/(4n²) ) ] / (1 + z²/n)

10.7 Reproducibility

The study log records:

• grading model with version and date, program build;
• random seed, normalisation and pseudonymisation;
• gold-standard documents;
• per answer the origin, question and SHA-256 checksum.

The full wording of the grading request is attached to the report. An AI grading is not deterministic; repeated runs on the same answers show the variability of the grading itself.

10.8 What the program does not compute – recommendations for confirmatory studies

• No significance test between conditions. Differences between Reporter, control and imported answers are descriptive. For a confirmatory study, register the hypothesis in advance and test pairwise per case:
  • two conditions with a binary outcome (e.g. correct / not correct): McNemar's test [17];
  • more than two conditions: Cochran's Q [4];
  • ordinal grid levels: Wilcoxon signed-rank test [20] or Friedman test [9];
  • several tests: Holm's correction [11].
• Dependence of observations. Several answers per case and several reviewers per answer are not independent. The blinding check pools all guesses. For inference across cases, use mixed models with random effects for case and reviewer.
• Small samples. Since version 3.3 the blinding check computes exactly; with few guesses its significance nonetheless remains low – below eight guesses the program draws no conclusion.
• Sample size. The width of the Wilson intervals shows the precision. Plan the number of cases in advance from the desired interval width or power.

10.9 References

1. Bang H, Ni L, Davis CE. Assessment of blinding in clinical trials. Controlled Clinical Trials 2004;25(2):143–156. doi:10.1016/j.cct.2003.10.016
2. Bossuyt PM, Reitsma JB, Bruns DE et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 2015;351:h5527. doi:10.1136/bmj.h5527
3. Brown LD, Cai TT, DasGupta A. Interval estimation for a binomial proportion. Statistical Science 2001;16(2):101–133. doi:10.1214/ss/1009213286
4. Cochran WG. The comparison of percentages in matched samples. Biometrika 1950;37(3/4):256–266. doi:10.1093/biomet/37.3-4.256
5. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20(1):37–46. doi:10.1177/001316446002000104
6. Cohen JF, Korevaar DA, Altman DG et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open 2016;6(11):e012799. doi:10.1136/bmjopen-2016-012799
7. Conover WJ. Practical Nonparametric Statistics. 3rd ed. New York: Wiley 1999 (section 3.1, binomial test).
8. Durstenfeld R. Algorithm 235: Random permutation. Communications of the ACM 1964;7(7):420. doi:10.1145/364520.364540
9. Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 1937;32(200):675–701. doi:10.1080/01621459.1937.10503522
10. Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 2007;1(1):77–89. doi:10.1080/19312450709336664
11. Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979;6(2):65–70.
12. James KE, Bloch DA, Lee KK, Kraemer HC, Fuller RK. An index for assessing blindness in a multi-centre clinical trial: disulfiram for alcohol cessation – a VA cooperative study. Statistics in Medicine 1996;15(13):1421–1434.
13. Knuth DE. The Art of Computer Programming, Vol. 2: Seminumerical Algorithms. 3rd ed. Reading, MA: Addison-Wesley 1997 (sections 3.2.2 and 3.4.2).
14. Krippendorff K. Computing Krippendorff’s Alpha-Reliability. Annenberg School for Communication, Departmental Papers 43, 2011.
15. Krippendorff K. Content Analysis: An Introduction to Its Methodology. 4th ed. Thousand Oaks: SAGE 2019.
16. Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies 2025;22:216–242. doi:10.1111/jels.12413
17. McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 1947;12(2):153–157. doi:10.1007/BF02295996
18. Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine 1998;17(8):857–872.
19. Panickssery A, Bowman SR, Feng S. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. arXiv:2404.13076
20. Wilcoxon F. Individual comparisons by ranking methods. Biometrics Bulletin 1945;1(6):80–83. doi:10.2307/3001968
21. Wilson EB. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 1927;22(158):209–212. doi:10.1080/01621459.1927.10502953
22. University grading grid for theses (Assurance of Learning), guideline, as of 31 Jan 2025 (German).
23. Zheng L, Chiang WL, Sheng Y et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023, Datasets and Benchmarks Track. arXiv:2306.05685

11Documentation and reproducibility

Providers change their models constantly; without precise details AI studies are not reproducible. The report therefore records per answer: condition, provider, model ID, version (for imports e.g. the system fingerprint from the JSON export, for the Reporter the program build), date and time, the question (identical or different), the archive entry and a SHA-256 checksum of the imported file – so it can later be shown that exactly this answer was graded. Plus the grading model, the random seed, the settings for normalisation, alignment and pseudonymisation and the full wording of the grading requests. The log is also kept in the study folder.

12Data protection and consent

Answers by students or experts are personal data under the GDPR. The program considers this from the outset:

• Answers by humans are imported only with confirmed consent; table rows with “consent: no” are rejected.
• The button Consent (template) produces an adaptable consent form (purpose, data, pseudonymisation, transfer to the AI provider, retention, withdrawal, Art. 6(1)(a) and Art. 7 GDPR) – a starting point for study lead and ethics committee, not legal advice.
• No dependence. The template states: participation, non-participation and withdrawal have no influence on grades, exams or studies. Consent is not collected by the person who examines or grades, and the study lead does not learn who takes part until grading is complete.
• Participants are kept under an ID (T01, T02 … in the report); names do not belong in the ID. Normalisation removes letterhead and signature, pseudonymisation replaces names in the text, before anything goes to an AI.
• The assignment code → origin lies only in the study lead's key folder; the blind folder contains nothing that points to persons.

13The guided procedure

The study mode guides in seven steps from the call to the result. On the left the steps, on the right one page per step with an explanation and a graphic showing what happens there.

1. What is there? A readiness check with traffic lights: AI key, case in the archive, area of law, optionally gold standard, finished answers and reviewers. For every item it says what is missing and where it belongs – and it can be fixed right there, without changing windows: enter and test the AI key, choose the area of law, add expert documents or drag them onto the window, enter an own question, import answers – and below, which test is possible. If nothing is missing, the procedure starts directly with the case.
2. Case: an archive entry whose facts are shown – or an own question, for instance the question of other AI results. It is the question for all answers.
3. Gold standard (optional): the graphic shows where to file it (program folder › Kategorien › area of law › goldstandard), next to it the documents found, "Open folder" and "Reload". Without a gold standard the test runs against the sources; category and degree of agreement are omitted, and the page says what is measured.
4. Answers: Reporter runs, control without pre-processing, imported answers. If the gold standard is attached only to the Reporter runs, the program points out the unfair comparison.
5. Blinding: normalise, align linguistically, blind folder – every option explained.
6. STARD: optionally a yes/no target question and the reference (from the gold standard or set by the study lead).
7. Overview & start: what is compared, how the test runs, how many AI requests it needs. "Start test" works in the background; the progress bar shows every step and has a Cancel button that ends the whole test cleanly.

At the end the result window opens: the key figures, with a gold standard per condition a bar with the shares correct, incomplete and hallucinated and the agreement with the gold standard, below the next steps as buttons – open report, study folder, import ratings, STARD report.

14State of research and market

The most important reference is the Stanford study by Magesh, Surani, Dahl, Suzgun, Manning and Ho, “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools”, Journal of Empirical Legal Studies 22 (2025), 216–242: legal RAG (Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI) reduces hallucinations, but they remain substantial – about 17 to 33 percent depending on the product. The study answered vendor claims that RAG eliminates hallucinations and was preregistered. The group also describes how laborious the manual grading was – every research group so far builds its own grading process, mostly with spreadsheets and a lot of manual work. Exactly this work becomes easier with the study mode; such studies are not competition but the kind of work the tool is built for.

Finer benchmarks such as ClaimRAG-LAW (Das, Abualhaija and Bianculli; 317 question-answer pairs, 968 expert-validated claims, separate evaluation of retrieval and generation down to individual claims) and a study on Austrian VAT law (Luketina, Benkel and Schuetz, Journal of Business Analytics 2026: fine-tuning and RAG on textbook and firm cases – properly configured a support for professionals, not ready for full automation) show the direction. There are also general evaluation platforms for developers and providers of legal benchmarks: the Vals AI Legal Research Report (14 October 2025; 200 US research questions, 199 evaluated) found, in the weighted overall score (accuracy 50%, authoritativeness 40%, appropriateness 10%), the specialised tools at 76 to 78 percent (Counsel Stack 78, Alexi 76, Midpage 76), ChatGPT at 74 percent and lawyers at 69 percent – a general model almost level with specialised products. The large research platforms did not take part in this study. The BenGER platform combines task creation, collaborative annotation, model runs and evaluation for German law. Such solutions address developers and researchers and rather test from outside. And with GDPval OpenAI showed that blinded expert comparisons between AI and human work are the recognised approach – as a one-off study.

The distinction: the study mode addresses users and decision-makers – professionals who want to know whether a technology works for their cases before they deploy or buy it. So the solutions compete less than it first appears. The unique selling point is formulated through the overall package and the target group, not through individual features.

All figures in this chapter were checked against the original sources on 5 October 2026 (Stanford study: JELS 22(2), 216–242, doi:10.1111/jels.12413 – Lexis+ AI 17%, Westlaw AI-Assisted Research 33%, Ask Practical Law AI 17% hallucinations, preregistered with the Open Science Framework; ClaimRAG-LAW: arXiv:2605.21071; Vals AI: original report of 14 October 2025, weighted overall score; GDPval: 1,320 tasks from 44 occupations, blinded pairwise comparisons by experts of the relevant occupation, arXiv:2510.04374); the study on Austrian VAT law (arXiv:2507.08468) on 6 October 2026.

15Next steps

• Targeted research whether a comparable tool exists after all – for instance from a university project; such solutions are often little known. Only then is the unique selling point communicated externally.
• Independent expert review of method, grading catalogue and instructions by professionals not involved in the project.
• Real cases from Austrian law with several expert documents per case instead of the invented test case so far.
• Validation of the AI grading against human reviewers – with the new measure from ‘Import ratings’.
• Pilot study with external users – two or three – for instance a chair of civil or public law or a larger law firm. A real result produced with the tool convinces more than any feature list.
• Preregistration of the study design before the evaluation – case collection, gold standard, categories, measures, analysis plan –, as the Stanford group did. As of this version nothing has been preregistered yet; a public preregistration is planned, for instance with the Open Science Framework.

16Limits

• Checking rules today. Existence, date and origin of cited decisions are checked automatically. The version of provisions, the comparison with the headnote, format check, foreign legal orders and legal remedies are not yet checked automatically; invented facts are measured by the study mode, not in running use.
• An AI pre-sorts answers in the study mode. That creates order and comparability but does not replace expert review – that is what the blind folder is for, and how well the AI agrees with the humans is measured.
• Narrow evidence base. The preliminary results so far rest on one invented case, one model (Claude Opus 4.8) and one test day (1 October 2026). They show that the tool works – not how good a technology is in general. Every external presentation states this just as clearly.
• With a gold standard the comparison is only as good as the gold standard. Expert documents should solve the case fully and name the key sources.
• Normalisation removes outward features; style and line of argument can still reveal the origin. Whether the blinding held is shown by the blinding check.
• Scanned handwriting is not read; PDF and Word formats need Microsoft Word on the computer.
• Cost: every Reporter and control run costs as much as one assessment, plus the grading request and, where applicable, the retrieval stage and the alignment.

17What has been dropped

The former study procedure – arms, model solution, reviewer sheets, series run, repetition with frozen sources and the direct comparison in three versions – has been removed from the program. It was too complicated and measured AI versions against each other instead of against professional knowledge. The first evaluation from that time remains documented in the dossier Comparison evaluation. Since version 3.4 the gold standard is no longer the main yardstick but an optional scientific add-on; a benchmark from published decisions is not used as a yardstick (chapter 1).