Coverage for orchestrant/benchmark/determinism.py: 100%

1"""Does this lane sample at temperature 0? The evidence bench_compare trusts. 

2 

3A report's `determinism_probe` decides how bench_compare reads it: a lane 

4recorded deterministic has its single-draw flips judged strictly, as real 

5regressions. So a change to this probe changes what a report CLAIMS, and the 

6tools that run it (bench_tools, bench_coding) put this file in their 

7`tool_sha256`. 

8 

9It used to live in provenance.py, and those tools fingerprinted that whole 

10module for its sake — so every edit to the provenance plumbing (a new field, a 

11runtime-detection fix) read on the next comparison as "BENCHMARK SOURCE 

12CHANGED — a score difference may be the grader", while the grader was 

13provably untouched. provenance re-exports these names, so existing imports 

14keep working. 

15""" 

16 

17import hashlib 

18 

19 

20PROBE_PROMPT = "Write one sentence about the sea." 

21 

22 

23SPACER_PROMPT = "Reply with the single word: ok" 

24 

25 

26def determinism_probe(base_url, model, post, prompt=PROBE_PROMPT, max_tokens=48): 

27 """Send the same request twice, with another between; did the outputs match? 

28 

29 `post(url, payload) -> dict` is injected so this can run without a server 

30 (tests) and so callers pick the transport. "Deterministic" here means two 

31 draws at temperature 0 agreed byte-for-byte — evidence, not proof, and it is 

32 recorded as such so a --repeats 1 flip can be read for what it is. 

33 

34 The prompt must leave the model real choices. The first version asked for 

35 "the single word: ready" in 8 tokens — an answer with almost no entropy, 

36 which a SAMPLING lane repeats verbatim. On GenieX v0.6.1 it recorded the 

37 QAIRT lane as deterministic while two open-ended requests at temperature 0 

38 came back different, and bench_compare then read that lane's single-draw 

39 flips as real regressions. 

40 

41 The two draws are NOT sent back to back. On GenieX (v0.6.1 and v0.7.0, 

42 measured 2026-09-24) an identical request sent twice in a row takes a cache 

43 path that changes the reply on both lanes — llama.cpp prefills 0 tokens and 

44 samples the first token from the previous reply's logits; QAIRT reuses 

45 part of the dialog — while after any other request both answer as if cold. 

46 A spacer request between the draws measures the sampler, not that bug. 

47 """ 

48 payload = { 

49 "model": model, 

50 "temperature": 0, 

51 "max_tokens": max_tokens, 

52 "stream": False, 

53 "messages": [{"role": "user", "content": prompt}], 

54 } 

55 spacer = { 

56 **payload, 

57 "max_tokens": 1, 

58 "messages": [{"role": "user", "content": SPACER_PROMPT}], 

59 } 

60 outputs = [] 

61 try: 

62 for i in range(2): 

63 if i: 

64 post(f"{base_url}/v1/chat/completions", spacer) 

65 reply = post(f"{base_url}/v1/chat/completions", payload) 

66 content = ( 

67 (reply.get("choices") or [{}])[0].get("message", {}).get("content") 

68 ) 

69 if content is None: 

70 raise ValueError("reply carried no message content") 

71 outputs.append(content) 

72 except Exception as e: 

73 return { 

74 "deterministic": None, 

75 "requests": len(outputs), 

76 "prompt": prompt, 

77 "error": f"{type(e).__name__}: {e}"[:200], 

78 } 

79 return { 

80 "deterministic": outputs[0] == outputs[1], 

81 "requests": 3, 

82 "spacer": True, 

83 "prompt": prompt, 

84 "output_sha256": [hashlib.sha256(o.encode()).hexdigest()[:16] for o in outputs], 

85 "error": None, 

86 }