feat(live): validate and swap compiled shaders between guest frames

commit: 76b7f307a15ad03540da2628a3e26ef75bbc1bef browse tree
parent: 7cb596d931d773db10cc8df8c329cd8b881a8826
author: james <j@cpl0.zip>
signature: Verified ~james signed with a key registered by ~james
date: 2026-11-01 04:40 UTC

13 files changed, +997 −22

Makefile modified +5 −1
README.md modified +19 −3
ROOT modified +2 −0
cli/driver.ML modified +3 −1
cli/watch.py added +151 −0
guest/Remy.HC modified +221 −13
src/Bounds.thy added +116 −0
src/Delivery.thy added +41 −0
src/Export.thy modified +2 −2
src/X86.thy modified +1 −1
tests/guest.py added +263 −0
tests/live.py added +172 −0
tests/native.py modified +1 −1

Makefile modified

@@ -1,6 +1,6 @@
1	1	ISABELLE ?= isabelle
2	2
3		-.PHONY: all check measure payload clean
	3	+.PHONY: all check measure payload guest clean
4	4
5	5	all: .build/ready
6	6

@@ -14,6 +14,7 @@ all: .build/ready
14	14	check: all
15	15	python3 tests/native.py
16	16	python3 tests/cli.py
	17	+ python3 tests/live.py
17	18
18	19	measure: all
19	20	python3 tests/measure.py

@@ -23,5 +24,8 @@ payload: all
23	24	@set -e; for name in Rings Weave Cells; do .build/remy examples/$$name.remy .build/payload/$$name.BIN; .build/remy vectors examples/$$name.remy .build/payload/$$name.DAT; cp examples/$$name.remy .build/payload/$$name.TXT; done
24	25	cp guest/Remy.HC .build/payload/Remy.HC
25	26
	27	+guest: payload
	28	+ python3 tests/guest.py "$(TEMPLE_ISO)"
	29	+
26	30	clean:
27	31	rm -rf .build

README.md modified

@@ -1,4 +1,4 @@
1		-Pixel expressions compile to native x86+64 for TempleOS through a compiler implemented in Isabelle/HOL with proofs of constant folding, lexical slot allocation, bounded stack execution and correspondence with a register and stack model
	1	+Pixel expressions compile to native x86-64 for TempleOS through an Isabelle/HOL compiler, with proofs of constant folding, register allocation, bounded branching and instruction-level execution of the emitted bytes
2	2
3	3	[![i like men](media/demo.gif)](media/demo.mp4)
4	4

@@ -13,7 +13,7 @@ It uses:
13	13	- Isabelle2026 and its bundled Poly/ML toolchain
14	14	- Python
15	15
16		-Copy these files onto `C:/` in TempleOS and run:
	16	+Copy `.build/payload/*` onto `C:/` in TempleOS and run:
17	17
18	18	```c
19	19	#include "C:/Remy.HC";

@@ -33,6 +33,22 @@ The byte interpreter follows encoded jumps and consumes a synthetic return addre
33	33
34	34	Both evaluations return `3`. `make measure` compares the original stack backend with the current compiler on the three examples and reports bytes, static stack instructions and median host pixel-kernel time
35	35
	36	+## Live
	37	+
	38	+The HOL packet builder serialises native code, its source snapshot and 512 reference rows for the trusted local compiler. The guest checks lengths, a checksum and every reference result before switching shaders between frames
	39	+
	40	+`make guest` tests a copy of an official TempleOS RedSea ISO and leaves a bootable image that starts `Live(0)` automatically. Start that image with COM1 attached to a Unix socket, then edit the watched source file
	41	+
	42	+```sh
	43	+make guest TEMPLE_ISO=/path/to/TempleOS.ISO
	44	+qemu-system-x86_64 -m 512 -smp 2 -cdrom .build/guest.iso \
	45	+ -chardev socket,id=live,path=/tmp/remy.sock,server=on,wait=off \
	46	+ -serial chardev:live &
	47	+python3 cli/watch.py examples/Rings.remy --socket /tmp/remy.sock
	48	+```
	49	+
	50	+The watcher keeps one transfer in flight and coalesces edits while awaiting its acknowledgement. Syntax errors and rejected packets leave the current shader running, and Escape stops the guest renderer and receiver
	51	+
36	52	## Correctness
37	53
38	54	Lexical bindings share the evaluation stack with spilled intermediates. The [allocator](src/Allocation.thy) keeps one intermediate in a register and adjusts variable offsets when it must spill, while the [branching compiler](src/Branches.thy) preserves the incoming frame and returns the expression result

@@ -49,6 +65,6 @@ The [encoder](src/core/Encoding.thy) proves inversion and byte bounds for immedi
49	65
50	66	Arithmetic wraps modulo `2^64` and shifts mask their count to six bits
51	67
52		-Nonconstant conditionals use forward `jz` and `jmp` instructions so only the selected arm runs. The validator checks both arms for matching stack heights and protected input registers. The HolyC runner masks each result to four bits for the TempleOS palette
	68	+Nonconstant conditionals use forward `jz` and `jmp` instructions so only the selected arm runs. The validator checks both arms for matching stack heights and protected input registers, with a proved [512-word bound](src/Bounds.thy) above the caller frame. The HolyC runner masks each result to four bits for the TempleOS palette
53	69
54	70	The [x86 model](src/X86.thy) follows the [Intel instruction descriptions](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) for the emitted subset using a word-addressed stack. The proved function body preserves its incoming stack and pointer, then `ret` removes the return address and advances that pointer by eight bytes. Arithmetic flags that the compiler never reads are abstracted, and hardware faults, the HolyC bridge, extraction and the OS remain outside the proof

ROOT modified

@@ -16,7 +16,9 @@ session Remy in src = Remybase +
16	16	theories
17	17	Allocation
18	18	Branches
	19	+ Bounds
19	20	X86
20	21	Frontend
	22	+ Delivery
21	23	Examples
22	24	Export

cli/driver.ML modified

@@ -66,6 +66,8 @@ fun main () =
66	66	(case CommandLine.arguments () of
67	67	["vectors", source, destination] =>
68	68	(case Remy.vectors (read source) of (true, bytes) => write destination bytes \| result => output result)
	69	+ \| ["package", revision, source, destination] =>
	70	+ emit destination (Remy.delivery (chars revision) (read source))
69	71	\| ["eval", source, x, y, t] => output (Remy.reference (read source) (chars x) (chars y) (chars t))
70	72	\| ["simulate", binary, x, y, t] => output (Remy.simulate (read binary) (chars x) (chars y) (chars t))
71	73	\| ["limits", depth, nodes, stack, source, destination] =>

@@ -74,7 +76,7 @@ fun main () =
74	76	\| ["disassemble", binary] => output (Remy.disassemble (read binary))
75	77	\| ["baseline", source, destination] => emit destination (Remy.baseline (read source))
76	78	\| [source, destination] => emit destination (Remy.artifact (read source))
77		- \| _ => raise Fail "usage: remy check binary \| remy disassemble binary \| remy simulate binary x y t \| remy baseline source binary \| remy limits depth nodes stack source binary \| remy source binary \| remy vectors source data \| remy eval source x y t (source - reads standard input)")
	79	+ \| _ => raise Fail "usage: remy check binary \| remy disassemble binary \| remy simulate binary x y t \| remy package revision source packet \| remy baseline source binary \| remy limits depth nodes stack source binary \| remy source binary \| remy vectors source data \| remy eval source x y t (source - reads standard input)")
78	80	handle exn =>
79	81	(TextIO.output (TextIO.stdErr, "remy: " ^ General.exnMessage exn ^ "\n");
80	82	OS.Process.exit OS.Process.failure);

cli/watch.py added

@@ -0,0 +1,151 @@
	1	+import argparse
	2	+import os
	3	+from pathlib import Path
	4	+import secrets
	5	+import socket
	6	+import struct
	7	+import subprocess
	8	+import sys
	9	+import tempfile
	10	+import time
	11	+
	12	+
	13	+ROOT = Path(__file__).resolve().parents[1]
	14	+MAGIC = b'RMA1'
	15	+
	16	+
	17	+def snapshot(path):
	18	+ with path.open('rb') as stream:
	19	+ before = os.fstat(stream.fileno())
	20	+ data = stream.read(65537)
	21	+ after = path.stat()
	22	+ if (before.st_ino, before.st_size, before.st_mtime_ns) != (after.st_ino, after.st_size, after.st_mtime_ns):
	23	+ return None
	24	+ if len(data) > 65536:
	25	+ raise ValueError('source exceeds 64 KiB')
	26	+ return data
	27	+
	28	+
	29	+def compile_packet(compiler, source, revision, directory):
	30	+ path, packet = directory / 'source', directory / 'packet'
	31	+ path.write_bytes(source)
	32	+ with subprocess.Popen([compiler, 'package', str(revision), path, packet],
	33	+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) as process:
	34	+ try:
	35	+ _, error = process.communicate(timeout=30)
	36	+ except BaseException:
	37	+ process.kill()
	38	+ process.communicate()
	39	+ raise
	40	+ if process.returncode:
	41	+ raise ValueError(error.strip())
	42	+ return packet.read_bytes()
	43	+
	44	+
	45	+def watch(source, endpoint, compiler, interval):
	46	+ previous = None
	47	+ packet = None
	48	+ connection = None
	49	+ flight = None
	50	+ completed = None
	51	+ sent = 0.0
	52	+ revision = secrets.randbelow((1 << 32) - 1) + 1
	53	+ replies = bytearray()
	54	+ next_connection = 0.0
	55	+ source_error = None
	56	+ with tempfile.TemporaryDirectory() as directory:
	57	+ directory = Path(directory)
	58	+ try:
	59	+ while True:
	60	+ try:
	61	+ data = snapshot(source)
	62	+ source_error = None
	63	+ if data is not None and data != previous:
	64	+ previous = data
	65	+ candidate = revision % ((1 << 32) - 1) + 1
	66	+ try:
	67	+ compiled = compile_packet(compiler, data, candidate, directory)
	68	+ packet, revision = compiled, candidate
	69	+ print(f'revision {revision}: compiled', flush=True)
	70	+ except (ValueError, subprocess.TimeoutExpired) as error:
	71	+ print(error, file=sys.stderr, flush=True)
	72	+ except (OSError, ValueError) as error:
	73	+ message = str(error)
	74	+ if message != source_error:
	75	+ print(message, file=sys.stderr, flush=True)
	76	+ source_error = message
	77	+ try:
	78	+ if connection is None:
	79	+ if time.monotonic() < next_connection:
	80	+ time.sleep(interval)
	81	+ continue
	82	+ connection = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
	83	+ connection.settimeout(1)
	84	+ connection.connect(str(endpoint))
	85	+ connection.settimeout(interval)
	86	+ flight = None
	87	+ completed = None
	88	+ sent = 0.0
	89	+ replies.clear()
	90	+ if packet is not None and flight is None and revision != completed:
	91	+ connection.settimeout(180)
	92	+ connection.sendall(packet)
	93	+ connection.settimeout(interval)
	94	+ flight = revision
	95	+ sent = time.monotonic()
	96	+ if flight is not None and time.monotonic() - sent > 180:
	97	+ raise TimeoutError('guest acknowledgement timed out')
	98	+ try:
	99	+ chunk = connection.recv(4096)
	100	+ if not chunk:
	101	+ raise ConnectionError('guest disconnected')
	102	+ replies.extend(chunk)
	103	+ except socket.timeout:
	104	+ pass
	105	+ while len(replies) >= 16:
	106	+ if replies[:4] != MAGIC:
	107	+ del replies[0]
	108	+ continue
	109	+ _, received, status, active = struct.unpack('<4sIII', replies[:16])
	110	+ del replies[:16]
	111	+ if received == 0 and status == 0:
	112	+ completed = None
	113	+ flight = None
	114	+ elif received == flight:
	115	+ completed = received
	116	+ flight = None
	117	+ print(f'revision {received}: ' + ('accepted' if status == 1 else f'rejected ({status}); active {active}'),
	118	+ flush=True)
	119	+ except OSError as error:
	120	+ if connection is not None:
	121	+ connection.close()
	122	+ if sent:
	123	+ print(str(error), file=sys.stderr, flush=True)
	124	+ next_connection = time.monotonic() + (6 if sent else interval)
	125	+ connection = None
	126	+ sent = 0.0
	127	+ time.sleep(interval)
	128	+ finally:
	129	+ if connection is not None:
	130	+ connection.close()
	131	+
	132	+
	133	+def main():
	134	+ parser = argparse.ArgumentParser()
	135	+ parser.add_argument('source', type=Path)
	136	+ parser.add_argument('--socket', required=True, type=Path)
	137	+ parser.add_argument('--compiler', type=Path, default=ROOT / '.build/remy')
	138	+ parser.add_argument('--interval', type=float, default=0.25)
	139	+ args = parser.parse_args()
	140	+ if not 0.01 <= args.interval <= 10:
	141	+ parser.error('interval must be between 0.01 and 10 seconds')
	142	+ if not args.compiler.is_file() or not os.access(args.compiler, os.X_OK):
	143	+ parser.error('build an executable compiler with make first')
	144	+ try:
	145	+ watch(args.source, args.socket, args.compiler.resolve(), args.interval)
	146	+ except KeyboardInterrupt:
	147	+ pass
	148	+
	149	+
	150	+if __name__ == '__main__':
	151	+ main()

guest/Remy.HC modified

@@ -18,19 +18,171 @@ CDC *image=NULL;
18	18	U8 *shader=NULL;
19	19	U8 *program=NULL;
20	20	I64 selected;
	21	+U8 *directory="C:/";
	22	+U8 *incoming=NULL;
	23	+I64 received=0,expected=0;
	24	+U32 synchronise=0,revision=0;
	25	+I64 display_lock=0,reply_lock=0,transfer=0;
	26	+CTask *receiver=NULL;
	27	+Bool live_mode=FALSE;
	28	+F64 last_received;
21	29
22		-U0 Verify()
	30	+U0 Reply(U32 sequence,U32 status)
	31	+{
	32	+ U32 words[4];
	33	+ I64 i;
	34	+ F64 until;
	35	+ U8 *bytes=words;
	36	+ while (LBts(&reply_lock,0)) Yield;
	37	+ words[0]=0x31414D52;
	38	+ words[1]=sequence;
	39	+ words[2]=status;
	40	+ words[3]=revision;
	41	+ for (i=0;i<16;i++) {
	42	+ until=tS()+0.1;
	43	+ while (!(InU8(0x3FD)&0x20)) {
	44	+ if (tS()>until) {
	45	+ LBtr(&reply_lock,0);
	46	+ return;
	47	+ }
	48	+ }
	49	+ OutU8(0x3F8,bytes[i]);
	50	+ }
	51	+ LBtr(&reply_lock,0);
	52	+}
	53	+
	54	+U64 Checksum(U8 *bytes,I64 size,U64 value=14695981039346656037)
	55	+{
	56	+ I64 i;
	57	+ for (i=0;i<size;i++) value=(value^bytes[i])*1099511628211;
	58	+ return value;
	59	+}
	60	+
	61	+U0 Accept()
	62	+{
	63	+ U32 *header=incoming;
	64	+ U64 sum,*rows;
	65	+ U8 next_code=NULL,next_source=NULL,old_code,old_source;
	66	+ I64 i;
	67	+ Bool matches=TRUE;
	68	+ sum=Checksum(incoming,20);
	69	+ sum=Checksum(incoming+28,expected-28,sum);
	70	+ if (sum!=(incoming+20)(U64 )) {
	71	+ Reply(header[4],2);
	72	+ return;
	73	+ }
	74	+ if (header[4]==revision) {
	75	+ Reply(header[4],1);
	76	+ return;
	77	+ }
	78	+ try {
	79	+ next_code=MAlloc(header[1]);
	80	+ next_source=CAlloc(header[2]+1);
	81	+ MemCpy(next_code,incoming+28,header[1]);
	82	+ MemCpy(next_source,incoming+28+header[1],header[2]);
	83	+ rows=(incoming+28+header[1]+header[2])(U64 *);
	84	+ for (i=0;i<512;i++) {
	85	+ if (Pixel(next_code,rows[4i],rows[4i+1],rows[4i+2])!=rows[4i+3]) {
	86	+ matches=FALSE;
	87	+ break;
	88	+ }
	89	+ }
	90	+ if (matches) {
	91	+ while (LBts(&display_lock,0)) Yield;
	92	+ old_code=shader;
	93	+ old_source=program;
	94	+ shader=next_code;
	95	+ program=next_source;
	96	+ next_code=NULL;
	97	+ next_source=NULL;
	98	+ revision=header[4];
	99	+ LBtr(&display_lock,0);
	100	+ Free(old_code);
	101	+ Free(old_source);
	102	+ Reply(header[4],1);
	103	+ } else Reply(header[4],3);
	104	+ Free(next_code);
	105	+ Free(next_source);
	106	+ } catch {
	107	+ Free(next_code);
	108	+ Free(next_source);
	109	+ Reply(header[4],3);
	110	+ }
	111	+}
	112	+
	113	+U0 Poll()
	114	+{
	115	+ I64 i;
	116	+ U64 byte;
	117	+ F64 until;
	118	+ U32 *header=incoming;
	119	+ if (received && tS()-last_received>5) {
	120	+ received=0;
	121	+ expected=0;
	122	+ synchronise=0;
	123	+ }
	124	+ if (!(InU8(0x3FD)&1)) return;
	125	+ until=tS()+0.004;
	126	+ for (i=0;i<8192;) {
	127	+ if (!(InU8(0x3FD)&1)) {
	128	+ if (tS()>=until) break;
	129	+ goto skip;
	130	+ }
	131	+ byte=InU8(0x3F8)&255;
	132	+ i++;
	133	+ last_received=tS();
	134	+ if (received<4) {
	135	+ synchronise=(synchronise>>8)\|(byte(U32)<<24);
	136	+ if (synchronise==0x31594D52) {
	137	+ header[0]=synchronise;
	138	+ received=4;
	139	+ }
	140	+ goto skip;
	141	+ }
	142	+ incoming[received++]=byte;
	143	+ if (received==28) {
	144	+ if (!header[1] \|\| header[1]>65536 \|\| !header[2] \|\| header[2]>65536 \|\|
	145	+ header[3]!=512 \|\| !header[4]) {
	146	+ Reply(header[4],4);
	147	+ received=0;
	148	+ expected=0;
	149	+ synchronise=0;
	150	+ goto skip;
	151	+ }
	152	+ expected=28+header[1]+header[2]+16384;
	153	+ }
	154	+ if (expected && received==expected) {
	155	+ LBts(&transfer,0);
	156	+ break;
	157	+ }
	158	+skip:;
	159	+ }
	160	+}
	161	+
	162	+U0 Receive(U8 *)
	163	+{
	164	+ while (TRUE) {
	165	+ if (Bt(&transfer,0)) Sleep(1);
	166	+ else {
	167	+ Poll();
	168	+ if (!received) Sleep(1);
	169	+ else Yield;
	170	+ }
	171	+ }
	172	+}
	173	+
	174	+Bool Verify()
23	175	{
24	176	I64 i,j,size,total=0;
25	177	U8 *code=NULL,path[64];
26	178	U64 *rows=NULL;
27	179	try {
28	180	for (i=0;i<3;i++) {
29		- StrPrint(path,"C:/%s.BIN",names[i]);
	181	+ StrPrint(path,"%s%s.BIN",directory,names[i]);
30	182	code=FileRead(path,&size);
31	183	if (!code) throw('Code');
32	184	"%-6s %d bytes ",names[i],size;
33		- StrPrint(path,"C:/%s.DAT",names[i]);
	185	+ StrPrint(path,"%s%s.DAT",directory,names[i]);
34	186	rows=FileRead(path,&size);
35	187	if (!rows \|\| size!=16384) throw('Vectors');
36	188	for (j=0;j<size/32;j++) {

@@ -48,22 +200,38 @@ U0 Verify()
48	200	} catch {
49	201	Free(rows);
50	202	Free(code);
	203	+ "verification failed after %d cases\n",total;
	204	+ return FALSE;
51	205	}
52	206	"native results match HOL evaluation (%d cases)\n",total;
	207	+ return TRUE;
53	208	}
54	209
55	210	U0 Draw(CTask ,CDC dc)
56	211	{
57		- dc->color=BLUE;
58		- GrPrint(dc,80,24,"remy %s",names[selected]);
59		- dc->color=BLACK;
60		- GrBlot(dc,80,64,image);
61		- GrPrint(dc,80,416,"%s",program);
	212	+ U8 *title="live";
	213	+ if (!live_mode) title=names[selected];
	214	+ if (LBts(&display_lock,0)) return;
	215	+ try {
	216	+ dc->color=BLUE;
	217	+ GrPrint(dc,80,24,"remy %s revision %d",title,revision);
	218	+ dc->color=BLACK;
	219	+ GrBlot(dc,80,64,image);
	220	+ GrPrint(dc,80,416,"%s",program);
	221	+ LBtr(&display_lock,0);
	222	+ } catch {
	223	+ LBtr(&display_lock,0);
	224	+ }
62	225	}
63	226
64	227	U0 Release(Bool restore)
65	228	{
66	229	Fs->draw_it=NULL;
	230	+ if (receiver) {
	231	+ Kill(receiver);
	232	+ receiver=NULL;
	233	+ }
	234	+ while (LBts(&display_lock,0)) Yield;
67	235	if (restore) SettingsPop;
68	236	DCDel(image);
69	237	image=NULL;

@@ -71,30 +239,63 @@ U0 Release(Bool restore)
71	239	shader=NULL;
72	240	Free(program);
73	241	program=NULL;
	242	+ Free(incoming);
	243	+ incoming=NULL;
	244	+ LBtr(&display_lock,0);
74	245	}
75	246
76		-U0 Show(I64 which=0)
	247	+U0 Show(I64 which=0,Bool live=FALSE)
77	248	{
78		- I64 x,y,dy,t=0;
	249	+ I64 x,y,dy,t=0,cpu=0,key;
79	250	U8 color,path[64];
80	251	Bool restore=FALSE;
81	252	if (!(0<=which<3)) return;
82	253	try {
83	254	selected=which;
84		- StrPrint(path,"C:/%s.BIN",names[which]);
	255	+ live_mode=live;
	256	+ StrPrint(path,"%s%s.BIN",directory,names[which]);
85	257	shader=FileRead(path);
86	258	if (!shader) throw('Code');
87		- StrPrint(path,"C:/%s.TXT",names[which]);
	259	+ StrPrint(path,"%s%s.TXT",directory,names[which]);
88	260	program=FileRead(path);
89	261	if (!program) throw('Source');
90	262	image=DCNew(480,336);
	263	+ if (live) {
	264	+ incoming=CAlloc(147484);
	265	+ received=0;
	266	+ expected=0;
	267	+ synchronise=0;
	268	+ revision=0;
	269	+ OutU8(0x3F9,0);
	270	+ OutU8(0x3FB,0x80);
	271	+ OutU8(0x3F8,1);
	272	+ OutU8(0x3F9,0);
	273	+ OutU8(0x3FB,3);
	274	+ OutU8(0x3FA,7);
	275	+ OutU8(0x3FC,3);
	276	+ transfer=0;
	277	+ if (mp_cnt>1) cpu=1;
	278	+ receiver=Spawn(&Receive,NULL,"remy uart",cpu);
	279	+ if (!receiver) throw('Receiver');
	280	+ Reply(0,0);
	281	+ }
91	282	SettingsPush;
92	283	restore=TRUE;
93	284	AutoComplete(OFF);
94	285	WinMax;
95	286	DocClear;
	287	+ FlushMsgs;
96	288	Fs->draw_it=&Draw;
97		- while (!ScanChar) {
	289	+ while (TRUE) {
	290	+ key=ScanChar();
	291	+ if (key && (!live \|\| key==CH_ESC)) break;
	292	+ if (live && Bt(&transfer,0)) {
	293	+ Accept();
	294	+ received=0;
	295	+ expected=0;
	296	+ synchronise=0;
	297	+ LBtr(&transfer,0);
	298	+ }
98	299	for (y=0;y<112;y++) {
99	300	for (x=0;x<160;x++) {
100	301	color=Pixel(shader,x,y,t)&15;

@@ -107,8 +308,15 @@ U0 Show(I64 which=0)
107	308	Sleep(20);
108	309	}
109	310	} catch {
	311	+ PutExcept;
110	312	Release(restore);
	313	+ return;
111	314	}
112	315	Release(restore);
113	316	DocClear;
114	317	}
	318	+
	319	+U0 Live(I64 which=0)
	320	+{
	321	+ Show(which,TRUE);
	322	+}

src/Bounds.thy added

@@ -0,0 +1,116 @@
	1	+theory Bounds
	2	+ imports Branches
	3	+begin
	4	+
	5	+fun guarded_run :: "nat \<Rightarrow> opcode list \<Rightarrow> machine \<Rightarrow> bool" where
	6	+ "guarded_run limit [] s = (length (stack s) \<le> limit)"
	7	+\| "guarded_run limit (op # code) s = (length (stack s) \<le> limit \<and>
	8	+ (case transition op s of None \<Rightarrow> False \| Some q \<Rightarrow> guarded_run limit code q))"
	9	+
	10	+fun guarded_flow :: "nat \<Rightarrow> flow \<Rightarrow> machine \<Rightarrow> bool" where
	11	+ "guarded_flow limit (Basic code) s = guarded_run limit code s"
	12	+\| "guarded_flow limit (Sequence a b) s = (guarded_flow limit a s \<and>
	13	+ (case flowrun a s of None \<Rightarrow> False \| Some q \<Rightarrow> guarded_flow limit b q))"
	14	+\| "guarded_flow limit (Choice a b) s = guarded_flow limit (if registers s Rax = 0 then b else a) s"
	15	+
	16	+lemma effect_shift:
	17	+ "effect op n = Some m \<Longrightarrow> effect op (n + base) = Some (m + base)"
	18	+ by (cases op; cases n; auto split: if_splits; arith)
	19	+
	20	+lemma effect_frame:
	21	+ "effect op n = Some m \<Longrightarrow> transition op s = Some q \<Longrightarrow>
	22	+ drop m (stack q) = drop n (stack s)"
	23	+ by (cases op; cases n; cases "stack s") (auto simp: effect.simps transition.simps split: if_splits)
	24	+
	25	+lemma audit_limit:
	26	+ "n \<le> 512 \<Longrightarrow> audit code n = Some m \<Longrightarrow> m \<le> 512"
	27	+ by (induction code arbitrary: n) (auto split: if_splits bind_splits)
	28	+
	29	+lemma flowaudit_limit:
	30	+ "n \<le> 512 \<Longrightarrow> flowaudit code n = Some m \<Longrightarrow> m \<le> 512"
	31	+proof (induction code arbitrary: n m)
	32	+ case (Basic code)
	33	+ then show ?case by (auto dest: audit_limit)
	34	+next
	35	+ case (Sequence a b)
	36	+ obtain k where a: "flowaudit a n = Some k" and b: "flowaudit b k = Some m"
	37	+ using Sequence.prems(2) by (auto split: bind_splits)
	38	+ have "k \<le> 512" using Sequence.IH(1)[OF Sequence.prems(1) a] .
	39	+ then show ?case using Sequence.IH(2)[OF _ b] by blast
	40	+next
	41	+ case (Choice a b)
	42	+ have "flowaudit a n = Some m"
	43	+ using Choice.prems(2) by (auto split: option.splits if_splits)
	44	+ then show ?case using Choice.IH(1)[OF Choice.prems(1)] by blast
	45	+qed
	46	+
	47	+theorem audit_window:
	48	+ assumes "length (stack s) = base + n" "n \<le> 512" "audit code n = Some m"
	49	+ shows "\<exists>q. run code s = Some q \<and> length (stack q) = base + m \<and>
	50	+ drop m (stack q) = drop n (stack s) \<and> guarded_run (base + 512) code s"
	51	+ using assms
	52	+proof (induction code arbitrary: s n m)
	53	+ case Nil
	54	+ then show ?case by auto
	55	+next
	56	+ case (Cons op code)
	57	+ obtain k where effect: "effect op n = Some k" and bound: "k \<le> 512"
	58	+ and tail: "audit code k = Some m"
	59	+ using Cons.prems(3) by (auto split: if_splits bind_splits)
	60	+ have shifted: "effect op (length (stack s)) = Some (base + k)"
	61	+ using effect_shift[OF effect, of base] Cons.prems(1) by (simp add: add.commute)
	62	+ obtain p where p: "transition op s = Some p" "length (stack p) = base + k"
	63	+ using effect_sound[OF shifted] by blast
	64	+ obtain q where q: "run code p = Some q" "length (stack q) = base + m"
	65	+ "drop m (stack q) = drop k (stack p)" "guarded_run (base + 512) code p"
	66	+ using Cons.IH[OF p(2) bound tail] by blast
	67	+ have frame: "drop k (stack p) = drop n (stack s)"
	68	+ using effect_frame[OF effect p(1)] .
	69	+ have "length (stack s) \<le> base + 512" using Cons.prems(1,2) by arith
	70	+ then show ?case using p q frame by auto
	71	+qed
	72	+
	73	+theorem flow_window:
	74	+ assumes "length (stack s) = base + n" "n \<le> 512" "flowaudit code n = Some m"
	75	+ shows "\<exists>q. flowrun code s = Some q \<and> length (stack q) = base + m \<and>
	76	+ drop m (stack q) = drop n (stack s) \<and> inputs q = inputs s \<and>
	77	+ guarded_flow (base + 512) code s"
	78	+ using assms
	79	+proof (induction code arbitrary: s n m)
	80	+ case (Basic code)
	81	+ have audit: "audit code n = Some m" using Basic.prems by simp
	82	+ obtain q where q: "run code s = Some q" "length (stack q) = base + m"
	83	+ "drop m (stack q) = drop n (stack s)" "guarded_run (base + 512) code s"
	84	+ using audit_window[OF Basic.prems(1,2) audit] by blast
	85	+ show ?case using q audit_inputs[OF audit q(1)] by auto
	86	+next
	87	+ case (Sequence a b)
	88	+ obtain k where a: "flowaudit a n = Some k" and b: "flowaudit b k = Some m"
	89	+ using Sequence.prems(3) by (auto split: bind_splits)
	90	+ obtain p where p: "flowrun a s = Some p" "length (stack p) = base + k"
	91	+ "drop k (stack p) = drop n (stack s)" "inputs p = inputs s"
	92	+ "guarded_flow (base + 512) a s"
	93	+ using Sequence.IH(1)[OF Sequence.prems(1,2) a] by blast
	94	+ have bound: "k \<le> 512" using flowaudit_limit[OF Sequence.prems(2) a] .
	95	+ obtain q where q: "flowrun b p = Some q" "length (stack q) = base + m"
	96	+ "drop m (stack q) = drop k (stack p)" "inputs q = inputs p"
	97	+ "guarded_flow (base + 512) b p"
	98	+ using Sequence.IH(2)[OF p(2) bound b] by blast
	99	+ show ?case using p q by auto
	100	+next
	101	+ case (Choice a b)
	102	+ have a: "flowaudit a n = Some m" and b: "flowaudit b n = Some m"
	103	+ using Choice.prems(3) by (auto split: option.splits if_splits)
	104	+ show ?case
	105	+ using Choice.IH(1)[OF Choice.prems(1,2) a] Choice.IH(2)[OF Choice.prems(1,2) b]
	106	+ by (cases "registers s Rax = 0") auto
	107	+qed
	108	+
	109	+corollary checked_frame:
	110	+ "flowaudit code 0 = Some 0 \<Longrightarrow> \<exists>q. flowrun code s = Some q \<and>
	111	+ stack q = stack s \<and> inputs q = inputs s \<and>
	112	+ guarded_flow (length (stack s) + 512) code s"
	113	+ using flow_window[where s=s and base="length (stack s)" and n=0 and m=0 and code=code]
	114	+ by auto
	115	+
	116	+end

src/Delivery.thy added

@@ -0,0 +1,41 @@
	1	+theory Delivery
	2	+ imports Frontend
	3	+begin
	4	+
	5	+definition checksum :: "nat list \<Rightarrow> pixel" where
	6	+ "checksum bytes = foldl (\<lambda>h b. (h XOR word_of_nat b) * 1099511628211) 14695981039346656037 bytes"
	7	+
	8	+definition delivery :: "string \<Rightarrow> string \<Rightarrow> (bool \<times> string) \<times> string" where
	9	+ "delivery sequence source = (case (number sequence,prepare source) of
	10	+ (Some revision,Some e) \<Rightarrow>
	11	+ let emitted = artifact source; code = map nat_of_char (snd (fst emitted));
	12	+ data = code @ map nat_of_char source @ concat (map (row e) [0..<512]);
	13	+ header = [82,77,89,49] @ octets 4 (length code) @ octets 4 (length source) @
	14	+ octets 4 512 @ octets 4 revision in
	15	+ if fst (fst emitted) \<and> revision < 2 ^ 32 \<and> 0 < revision \<and>
	16	+ length code \<le> 65536 \<and> length source \<le> 65536
	17	+ then ((True,map char_of_nat (header @ octets 8 (unat (checksum (header @ data))) @ data)),snd emitted)
	18	+ else ((False,[]),''invalid live payload limits'' @ newline)
	19	+ \| (_,None) \<Rightarrow> ((False,[]),source_error source)
	20	+ \| _ \<Rightarrow> ((False,[]),''invalid live revision'' @ newline))"
	21	+
	22	+lemma checksum_empty: "checksum [] = 14695981039346656037"
	23	+ by (simp add: checksum_def)
	24	+
	25	+lemma checksum_step:
	26	+ "checksum (bytes @ [b]) = (checksum bytes XOR word_of_nat b) * 1099511628211"
	27	+ by (simp add: checksum_def foldl_append)
	28	+
	29	+
	30	+lemma row_size [simp]: "length (row e i) = 32"
	31	+ by (simp add: row_def)
	32	+
	33	+lemma rows_size [simp]: "length (concat (map (row e) xs)) = 32 * length xs"
	34	+ by (induction xs) auto
	35	+
	36	+theorem delivery_bounds:
	37	+ "delivery sequence source = ((True,packet),report) \<Longrightarrow>
	38	+ 16412 \<le> length packet \<and> length packet \<le> 147484"
	39	+ by (auto simp: delivery_def Let_def split: option.splits if_splits; linarith)
	40	+
	41	+end

src/Export.thy modified

@@ -1,11 +1,11 @@
1	1	theory Export
2		- imports Examples
	2	+ imports Examples Delivery
3	3	begin
4	4
5	5	definition ascii :: "integer \<Rightarrow> char" where "ascii = char_of"
6	6
7	7	definition ordinal :: "char \<Rightarrow> integer" where "ordinal = of_char"
8	8
9		-export_code validate disassemble configured artifact baseline vectors reference simulate ascii ordinal in SML module_name Remy file_prefix remy
	9	+export_code validate disassemble configured artifact baseline vectors reference simulate delivery ascii ordinal in SML module_name Remy file_prefix remy
10	10
11	11	end

src/X86.thy modified

@@ -1,5 +1,5 @@
1	1	theory X86
2		- imports Branches
	2	+ imports Bounds
3	3	begin
4	4
5	5	datatype micro = Imm64 pixel \| Imm32 pixel \| Copy register register \| Save register \| Restore register

tests/guest.py added

@@ -0,0 +1,263 @@
	1	+import json
	2	+from pathlib import Path
	3	+import re
	4	+import signal
	5	+import socket
	6	+import struct
	7	+import subprocess
	8	+import sys
	9	+import tempfile
	10	+import time
	11	+
	12	+from live import message
	13	+
	14	+
	15	+ROOT = Path(__file__).resolve().parents[1]
	16	+ENTRY = struct.Struct('<H38sqqQ')
	17	+
	18	+
	19	+def image(template, destination):
	20	+ data = bytearray(template.read_bytes())
	21	+ boots = [n for n in range(min(256, len(data) // 512))
	22	+ if data[n512+3] == 0x88 and data[n512+510:n*512+512] == b'\x55\xaa']
	23	+ if len(boots) != 1:
	24	+ raise ValueError('expected a TempleOS RedSea ISO')
	25	+ boot = boots[0]
	26	+ offset, sectors, root, bitmap, _ = struct.unpack_from('<5q', data, boot*512+8)
	27	+ if offset != boot or len(data) % 512:
	28	+ raise ValueError('invalid RedSea geometry')
	29	+ entries = []
	30	+ size = ENTRY.unpack_from(data, root*512)[3]
	31	+ for at in range(root512, root512+size, ENTRY.size):
	32	+ row = ENTRY.unpack_from(data, at)
	33	+ if row[1][0]:
	34	+ entries.append(row)
	35	+
	36	+ def append(payload):
	37	+ sector = len(data) // 512
	38	+ data.extend(payload)
	39	+ data.extend(bytes((-len(data)) % 512))
	40	+ return sector
	41	+
	42	+ files = {p.name: p.read_bytes() for p in (ROOT / '.build/payload').iterdir()}
	43	+ for name, payload in files.items():
	44	+ if len(name.encode('ascii')) >= 38:
	45	+ raise ValueError('RedSea filename exceeds 37 bytes')
	46	+ entries.append((0x800, name.encode('ascii'), append(payload), len(payload), 0))
	47	+ once = b'Cd("::/");;\n#include "Remy.HC"\ndirectory="::/";\nif (Verify()) Live(0);\n'
	48	+ once_sector = append(once)
	49	+ home = next(row for row in entries if row[1].split(b'\0')[0] == b'Home')
	50	+ home_start, home_size = home[2:4]
	51	+ for at in range(home_start512, home_start512+home_size, ENTRY.size):
	52	+ if not data[at+2]:
	53	+ data[at:at+ENTRY.size] = ENTRY.pack(0x800, b'Once.HC', once_sector, len(once), 0)
	54	+ break
	55	+ else:
	56	+ raise ValueError('template Home directory has no spare entry')
	57	+ new_root = len(data) // 512
	58	+ new_size = ((len(entries)ENTRY.size+511) // 512)512
	59	+ entries[0] = (entries[0][0], entries[0][1], new_root, new_size, entries[0][4])
	60	+ entries[1] = (entries[1][0], entries[1][1], new_root, 0, entries[1][4])
	61	+ append(b''.join(ENTRY.pack(*row) for row in entries))
	62	+ for row in entries[2:]:
	63	+ if row[0] & 0x10:
	64	+ parent = row[2]*512+ENTRY.size
	65	+ if ENTRY.unpack_from(data, parent)[1].split(b'\0')[0] == b'..':
	66	+ struct.pack_into('<q', data, parent+40, new_root)
	67	+ data.extend(bytes((-len(data)) % 2048))
	68	+ total = len(data) // 512
	69	+ if total - (offset + bitmap) >= bitmap5128:
	70	+ raise ValueError('template allocation bitmap is too small')
	71	+ struct.pack_into('<q', data, boot*512+16, total-offset)
	72	+ struct.pack_into('<q', data, boot*512+24, new_root)
	73	+ first = offset + bitmap
	74	+ for sector in range(offset+sectors, total):
	75	+ bit = sector-first
	76	+ data[(offset+1)*512+bit//8] \|= 1 << (bit % 8)
	77	+ for sector in (16,18):
	78	+ struct.pack_into('<I', data, sector*2048+80, len(data)//2048)
	79	+ struct.pack_into('>I', data, sector*2048+84, len(data)//2048)
	80	+ struct.pack_into('<I', data, sector*2048+152, new_root)
	81	+ struct.pack_into('>I', data, sector*2048+156, new_root)
	82	+ destination.write_bytes(data)
	83	+
	84	+
	85	+def exact(connection, count):
	86	+ result = bytearray()
	87	+ while len(result) < count:
	88	+ data = connection.recv(count-len(result))
	89	+ if not data:
	90	+ raise ConnectionError('guest serial connection closed')
	91	+ result.extend(data)
	92	+ return bytes(result)
	93	+
	94	+
	95	+def reply(connection):
	96	+ prefix = bytearray(exact(connection, 4))
	97	+ while prefix != b'RMA1':
	98	+ prefix.pop(0)
	99	+ prefix.extend(exact(connection, 1))
	100	+ return struct.unpack('<III', exact(connection, 12))
	101	+
	102	+
	103	+def packet(source, revision, directory):
	104	+ text, binary = directory / 'source', directory / 'packet'
	105	+ text.write_text(source)
	106	+ subprocess.run([ROOT / '.build/remy', 'package', str(revision), text, binary],
	107	+ check=True, capture_output=True, timeout=30)
	108	+ return binary.read_bytes()
	109	+
	110	+
	111	+def checksum(data):
	112	+ value = 14695981039346656037
	113	+ for byte in data:
	114	+ value = ((value ^ byte)*1099511628211) & ((1 << 64)-1)
	115	+ return value
	116	+
	117	+
	118	+def screen(qmp, path):
	119	+ qmp.write(json.dumps({'execute':'screendump','arguments':{'filename':str(path)}}).encode()+b'\n')
	120	+ while True:
	121	+ response = json.loads(qmp.readline())
	122	+ if 'return' in response:
	123	+ break
	124	+ if 'error' in response:
	125	+ raise RuntimeError(response)
	126	+ data = path.read_bytes()
	127	+ header = re.match(rb'P6\s+(\d+)\s+(\d+)\s+255\s', data)
	128	+ if not header:
	129	+ raise ValueError('invalid QEMU PPM screenshot')
	130	+ width, height = map(int, header.groups())
	131	+ assert width >= 560 and height >= 400
	132	+ pixels = data[header.end():]
	133	+ return b''.join(pixels[(ywidth+80)3:(ywidth+560)3] for y in range(64,400))
	134	+
	135	+
	136	+def main():
	137	+ if len(sys.argv) != 2 or not sys.argv[1]:
	138	+ raise SystemExit('usage: python3 tests/guest.py TempleOS.ISO')
	139	+ destination = ROOT / '.build/guest.iso'
	140	+ image(Path(sys.argv[1]), destination)
	141	+ with tempfile.TemporaryDirectory(prefix='remy') as directory:
	142	+ directory = Path(directory)
	143	+ serial, monitor = directory / 'serial', directory / 'qmp'
	144	+ log = (ROOT / '.build/guest.log').open('wb')
	145	+ process = subprocess.Popen(['qemu-system-x86_64','-m','512','-smp','2',
	146	+ '-accel','tcg,thread=multi','-cdrom',destination,'-boot','d','-display','none',
	147	+ '-monitor','none','-no-reboot','-chardev',f'socket,id=live,path={serial},server=on,wait=off',
	148	+ '-serial','chardev:live','-qmp',f'unix:{monitor},server=on,wait=off'],stdout=log,stderr=log)
	149	+ connection = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
	150	+ control = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
	151	+ qmp = None
	152	+ watcher = None
	153	+ try:
	154	+ until = time.monotonic()+15
	155	+ while not serial.exists() or not monitor.exists():
	156	+ if time.monotonic()>until or process.poll() is not None:
	157	+ raise RuntimeError('QEMU did not start')
	158	+ time.sleep(0.1)
	159	+ connection.connect(str(serial))
	160	+ connection.settimeout(90)
	161	+ control.connect(str(monitor))
	162	+ control.settimeout(5)
	163	+ qmp = control.makefile('rwb',buffering=0)
	164	+ json.loads(qmp.readline())
	165	+ qmp.write(b'{"execute":"qmp_capabilities"}\n')
	166	+ while 'return' not in json.loads(qmp.readline()):
	167	+ pass
	168	+ assert reply(connection) == (0,0,0)
	169	+ print('guest booted and verified all three payloads',flush=True)
	170	+ connection.settimeout(30)
	171	+ connection.sendall(packet('(+ x y)',1,directory))
	172	+ assert reply(connection) == (1,1,1)
	173	+ time.sleep(1)
	174	+ before = screen(qmp,ROOT / '.build/first.ppm')
	175	+ second = packet('(if (< x y) (+ x 3) (- y 9))',2,directory)
	176	+ connection.sendall(second)
	177	+ assert reply(connection) == (2,1,2)
	178	+ time.sleep(1)
	179	+ after = screen(qmp,ROOT / '.build/second.ppm')
	180	+ assert before != after, 'accepted revision did not change rendered pixels'
	181	+ print('guest changed rendered pixels after two verified revisions',flush=True)
	182	+ corrupt = bytearray(packet('(xor x y)',3,directory))
	183	+ corrupt[-1] ^= 1
	184	+ connection.sendall(corrupt)
	185	+ assert reply(connection) == (3,2,2)
	186	+ corrupt = bytearray(packet('(xor x y)',4,directory))
	187	+ corrupt[-1] ^= 1
	188	+ struct.pack_into('<Q',corrupt,20,checksum(corrupt[:20]+corrupt[28:]))
	189	+ connection.sendall(corrupt)
	190	+ assert reply(connection) == (4,3,2)
	191	+ time.sleep(1)
	192	+ assert screen(qmp,ROOT / '.build/rejected.ppm') == after
	193	+ invalid = ((1,0),(1,65537),(1,0xffffffff),(2,0),(2,65537),(3,0),(3,513),(4,0))
	194	+ for sequence,(field,value) in enumerate(invalid,10):
	195	+ header = bytearray(packet('(xor x y)',sequence,directory)[:28])
	196	+ struct.pack_into('<I',header,4*field,value)
	197	+ connection.sendall(header)
	198	+ assert reply(connection) == (0 if field == 4 else sequence,4,2)
	199	+ print('guest rejected checksum, vector and header failures without changing the active shader',flush=True)
	200	+ connection.sendall(second[:12])
	201	+ time.sleep(6)
	202	+ connection.sendall(packet('(xor x y)',0xffffffff,directory))
	203	+ assert reply(connection) == (0xffffffff,1,0xffffffff)
	204	+ print('guest recovered from a partial frame',flush=True)
	205	+ connection.close()
	206	+ source = directory/'watch'
	207	+ source.write_text('(* x y)')
	208	+ watcher = subprocess.Popen(['python3',ROOT/'cli/watch.py',source,'--socket',serial,'--interval','0.1'],
	209	+ stdout=subprocess.PIPE,stderr=subprocess.PIPE)
	210	+ message(watcher.stdout,b'accepted',timeout=90)
	211	+ time.sleep(1)
	212	+ watched = screen(qmp,ROOT / '.build/watched.ppm')
	213	+ source.write_text('(+ x)')
	214	+ message(watcher.stderr,b'syntax error',timeout=30)
	215	+ assert screen(qmp,ROOT / '.build/invalid.ppm') == watched
	216	+ source.write_text('(and x y)')
	217	+ message(watcher.stdout,b'accepted',timeout=90)
	218	+ time.sleep(1)
	219	+ final = screen(qmp,ROOT / '.build/final.ppm')
	220	+ assert final != watched
	221	+ print('watcher edits reached the real guest and an invalid source preserved its frame',flush=True)
	222	+ watcher.send_signal(signal.SIGINT)
	223	+ watcher.wait(timeout=5)
	224	+ assert watcher.returncode == 0
	225	+ qmp.write(b'{"execute":"human-monitor-command","arguments":{"command-line":"sendkey esc"}}\n')
	226	+ while 'return' not in json.loads(qmp.readline()):
	227	+ pass
	228	+ time.sleep(2)
	229	+ assert screen(qmp,ROOT / '.build/stopped.ppm') != final
	230	+ print('guest renderer and receiver stopped through Escape',flush=True)
	231	+ except Exception:
	232	+ if qmp is not None:
	233	+ try:
	234	+ screen(qmp,ROOT / '.build/failure.ppm')
	235	+ except Exception:
	236	+ pass
	237	+ raise
	238	+ finally:
	239	+ if watcher is not None:
	240	+ if watcher.poll() is None:
	241	+ watcher.terminate()
	242	+ try:
	243	+ watcher.wait(timeout=5)
	244	+ except subprocess.TimeoutExpired:
	245	+ watcher.kill()
	246	+ watcher.wait()
	247	+ watcher.stdout.close()
	248	+ watcher.stderr.close()
	249	+ connection.close()
	250	+ if qmp is not None:
	251	+ qmp.close()
	252	+ control.close()
	253	+ process.terminate()
	254	+ try:
	255	+ process.wait(timeout=10)
	256	+ except subprocess.TimeoutExpired:
	257	+ process.kill()
	258	+ process.wait()
	259	+ log.close()
	260	+
	261	+
	262	+if __name__ == '__main__':
	263	+ main()

tests/live.py added

@@ -0,0 +1,172 @@
	1	+import os
	2	+from pathlib import Path
	3	+import select
	4	+import signal
	5	+import socket
	6	+import struct
	7	+import subprocess
	8	+import tempfile
	9	+import time
	10	+
	11	+
	12	+ROOT = Path(__file__).resolve().parents[1]
	13	+HEADER = struct.Struct('<4sIIIIQ')
	14	+
	15	+
	16	+def checksum(data):
	17	+ value = 14695981039346656037
	18	+ for byte in data:
	19	+ value = ((value ^ byte)*1099511628211) & ((1 << 64)-1)
	20	+ return value
	21	+
	22	+
	23	+def exact(connection, count):
	24	+ data = bytearray()
	25	+ while len(data) < count:
	26	+ chunk = connection.recv(count-len(data))
	27	+ if not chunk:
	28	+ raise ConnectionError('watcher disconnected')
	29	+ data.extend(chunk)
	30	+ return bytes(data)
	31	+
	32	+
	33	+def receive(connection):
	34	+ head = exact(connection, HEADER.size)
	35	+ magic, code, source, rows, revision, digest = HEADER.unpack(head)
	36	+ assert magic == b'RMY1' and rows == 512 and 0 < revision < 1 << 32
	37	+ assert 0 < code <= 65536 and 0 < source <= 65536
	38	+ body = exact(connection, code+source+32*rows)
	39	+ assert digest == checksum(head[:20]+body)
	40	+ return head+body, body[code:code+source], revision
	41	+
	42	+
	43	+def acknowledge(connection, revision):
	44	+ response = struct.pack('<4sIII',b'RMA1',revision,1,revision)
	45	+ connection.sendall(response[:6])
	46	+ time.sleep(0.05)
	47	+ connection.sendall(response[6:])
	48	+
	49	+
	50	+def message(stream, expected, timeout=10):
	51	+ end = time.monotonic()+timeout
	52	+ data = b''
	53	+ while expected not in data:
	54	+ if not select.select([stream],[],[],max(0,end-time.monotonic()))[0]:
	55	+ raise AssertionError((expected,data))
	56	+ chunk = os.read(stream.fileno(),4096)
	57	+ if not chunk:
	58	+ raise AssertionError((expected,data))
	59	+ data += chunk
	60	+
	61	+
	62	+def main():
	63	+ with tempfile.TemporaryDirectory(prefix='remylive') as directory:
	64	+ directory = Path(directory)
	65	+ source, binary = directory/'source',directory/'packet'
	66	+ source.write_text('(+ x y)')
	67	+ subprocess.run([ROOT/'.build/remy','package','1',source,binary],check=True,capture_output=True)
	68	+ first = binary.read_bytes()
	69	+ magic, code, count, rows, revision, digest = HEADER.unpack(first[:28])
	70	+ assert (magic,count,rows,revision) == (b'RMY1',7,512,1)
	71	+ assert len(first) == 28+code+count+rows*32
	72	+ assert digest == checksum(first[:20]+first[28:])
	73	+ native = directory/'native'
	74	+ subprocess.run([ROOT/'.build/remy',source,native],check=True,capture_output=True)
	75	+ assert first[28:28+code] == native.read_bytes()
	76	+ vectors = first[28+code+count:]
	77	+ for at in range(0,len(vectors),32):
	78	+ x,y,t,result = struct.unpack_from('<4Q',vectors,at)
	79	+ assert result == (x+y) & ((1 << 64)-1)
	80	+ source.write_bytes(b'(+ x y);'+b'a'*(65536-8))
	81	+ subprocess.run([ROOT/'.build/remy','package',str((1 << 32)-1),source,binary],
	82	+ check=True,capture_output=True,timeout=30)
	83	+ maximum = binary.read_bytes()
	84	+ fields = HEADER.unpack(maximum[:28])
	85	+ assert fields[2] == 65536 and fields[4] == (1 << 32)-1
	86	+ assert fields[5] == checksum(maximum[:20]+maximum[28:])
	87	+ source.write_text('(+ x y)')
	88	+ for revision in ('0',str(1 << 32),'bad'):
	89	+ binary.write_bytes(b'preserve')
	90	+ result = subprocess.run([ROOT/'.build/remy','package',revision,source,binary],capture_output=True)
	91	+ assert result.returncode and binary.read_bytes() == b'preserve'
	92	+ source.write_text('(+ x)')
	93	+ result = subprocess.run([ROOT/'.build/remy','package','2',source,binary],capture_output=True)
	94	+ assert result.returncode and binary.read_bytes() == b'preserve'
	95	+ source.write_text('(+ x y)')
	96	+ server = socket.socket(socket.AF_UNIX,socket.SOCK_STREAM)
	97	+ endpoint = directory/'serial'
	98	+ server.bind(str(endpoint))
	99	+ server.listen(1)
	100	+ server.settimeout(15)
	101	+ watcher = subprocess.Popen(['python3',ROOT/'cli/watch.py',source,'--socket',endpoint,'--interval','0.05'],
	102	+ stdout=subprocess.PIPE,stderr=subprocess.PIPE)
	103	+ connection = None
	104	+ try:
	105	+ connection,_ = server.accept()
	106	+ connection.settimeout(10)
	107	+ original, text, revision = receive(connection)
	108	+ assert text == b'(+ x y)'
	109	+ acknowledge(connection,revision)
	110	+ message(watcher.stdout,b'accepted')
	111	+ source.write_text('(+ x)')
	112	+ message(watcher.stderr,b'syntax error')
	113	+ connection.settimeout(0.4)
	114	+ try:
	115	+ data = connection.recv(1)
	116	+ except socket.timeout:
	117	+ pass
	118	+ else:
	119	+ raise AssertionError(('invalid source produced output',data))
	120	+ connection.close()
	121	+ connection,_ = server.accept()
	122	+ connection.settimeout(10)
	123	+ resent,text,same = receive(connection)
	124	+ assert resent == original and same == revision
	125	+ acknowledge(connection,same)
	126	+ message(watcher.stdout,b'accepted')
	127	+ temporary = directory/'replacement'
	128	+ temporary.write_text('(- x y)')
	129	+ temporary.replace(source)
	130	+ updated,text,changed = receive(connection)
	131	+ assert text == b'(- x y)' and changed != revision
	132	+ message(watcher.stdout,b'compiled')
	133	+ source.write_text('(* x 3)')
	134	+ message(watcher.stdout,b'compiled')
	135	+ source.write_text('(* x 7)')
	136	+ message(watcher.stdout,b'compiled')
	137	+ connection.settimeout(0.3)
	138	+ try:
	139	+ data = connection.recv(1)
	140	+ except socket.timeout:
	141	+ pass
	142	+ else:
	143	+ raise AssertionError(('unacknowledged transfer was not coalesced',data))
	144	+ acknowledge(connection,changed)
	145	+ message(watcher.stdout,b'accepted')
	146	+ connection.settimeout(10)
	147	+ updated,text,changed = receive(connection)
	148	+ assert text == b'(* x 7)'
	149	+ acknowledge(connection,changed)
	150	+ message(watcher.stdout,b'accepted')
	151	+ connection.sendall(struct.pack('<4sIII',b'RMA1',0,0,0))
	152	+ reset,_,same = receive(connection)
	153	+ assert reset == updated and same == changed
	154	+ acknowledge(connection,same)
	155	+ finally:
	156	+ watcher.send_signal(signal.SIGINT)
	157	+ try:
	158	+ watcher.wait(timeout=5)
	159	+ except subprocess.TimeoutExpired:
	160	+ watcher.kill()
	161	+ watcher.wait()
	162	+ if connection is not None:
	163	+ connection.close()
	164	+ server.close()
	165	+ watcher.stdout.close()
	166	+ watcher.stderr.close()
	167	+ assert watcher.returncode == 0
	168	+ print('live packet bounds, independent vectors, compile rejection, fragmented replies, reconnect, atomic edits and guest restart passed')
	169	+
	170	+
	171	+if __name__ == '__main__':
	172	+ main()

tests/native.py modified

@@ -178,7 +178,7 @@ def main():
178	178	binary.write_bytes(b"preserve")
179	179	result = subprocess.run([ROOT / ".build/remy", path, binary], capture_output=True, timeout=30)
180	180	assert result.returncode != 0 and binary.read_bytes() == b"preserve", bad
181		- print(f"{len(cases)} programs, {comparisons} native comparisons, HOL reference checks and 11 rejection cases passed")
	181	+ print(f"{len(cases)} programs, {comparisons} native comparisons, byte interpreter and HOL reference checks, 11 rejection cases passed")
182	182
183	183
184	184	if __name__ == "__main__":

~james/remy

feat(live): validate and swap compiled shaders between guest frames

Makefile modified

README.md modified

ROOT modified

cli/driver.ML modified

cli/watch.py added

guest/Remy.HC modified

src/Bounds.thy added

src/Delivery.thy added

src/Export.thy modified

src/X86.thy modified

tests/guest.py added

tests/live.py added

tests/native.py modified