# Bulk Processing Notes for WWT Cubie Demo

## What was generated

- `blended_prompt_set_1500.jsonl`: human/audit workload file with source, NIST, Cubie, and expected Llama outcome fields.
- `litellm_batch_requests_1500_llama8b.jsonl`: upload-ready runtime JSONL for one Llama 8B batch job.
- `litellm_batches_llama8b/batch_01_llama8b.jsonl` through `batch_15_llama8b.jsonl`: 15 batch-split files with 100 requests each.
- `runtime_batch_manifest.csv`: counts, expected Cubie decisions, and same-model validation per split file.

## Runtime line shape

Each runtime line follows the OpenAI-compatible batch shape:

```json
{"custom_id":"wwt_0001","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-8b","temperature":0,"max_tokens":512,"metadata":{},"messages":[{"role":"system","content":"..."},{"role":"user","content":"..."}]}}
```

## LiteLLM batch submission pattern

```bash
curl http://localhost:4000/v1/files   -H "Authorization: Bearer $LITELLM_PROXY_KEY"   -F purpose="batch"   -F file="@litellm_batch_requests_1500_llama8b.jsonl"

curl http://localhost:4000/v1/batches   -H "Authorization: Bearer $LITELLM_PROXY_KEY"   -H "Content-Type: application/json"   -d '{"input_file_id":"<file-id>","endpoint":"/v1/chat/completions","completion_window":"24h","metadata":{"run_id":"cubie_wwt_zero_1500_llama8b"}}'
```

## Batch hard rules

1. Keep `custom_id` unique. Results may be returned out of order, so `custom_id` is the stable join key.
2. Keep one deployment/model per batch file/job. The Llama 70B follow-on must use a separate batch file/job.
3. Keep one endpoint per batch file. These files use `/v1/chat/completions` only.
4. Use UTF-8 JSONL with no byte-order mark.
5. Use `metadata.request_id`, `metadata.batch_id`, and `metadata.expected_reason_code` to join LiteLLM, Cubie, and telemetry logs.

## Cubie modes

Run the same runtime file in these modes:

1. Shadow mode: Cubie labels but does not block.
2. Baseline: Cubie bypassed or non-enforcing; all 1,500 requests reach the A100 path.
3. Cubie enforcement: Cubie prevents direct A100 work for deny/backoff/compress/review rows.
4. Replay enforcement: same request IDs, order, model, max_tokens, and temperature.

## Expected Llama flags

Every prompt includes these fields in the audit file and runtime metadata:

- `expected_llama_outcome`
- `expected_llama_finish_reason`
- `expected_llama_success_when_called`
- `expected_llama_failure_mode_without_cubie`
- `llama_issue_family`

These fields are intended to distinguish Cubie-gated preventable work from model-side Llama behavior such as context budget pressure, duplicate low-value completions, or validation errors.
