safety-research/bloom

Bloom is a scaffolded evaluation system which accepts as input an evaluation configuration (the "seed"), which specifies a target behavior (for example sycophancy, political bias or self-presevation), exemplary transcripts, and the types of interactions the user is interested in, and generates an evaluation suite of interactions with the target model that attempt to uncover the chosen behavior. As reflected in the name of the tool, the evaluation suite will grow differently depending on how it is seeded, which differs from other evaluations that may use a fixed elicitation technique and prompting pattern. Thus, for reproducibility, Bloom evaluations should always be cited together with their full seed configuration.

Results will appear in results/{behavior_name}/ including JSON files for each pipeline stage and individual transcript files.

This starts a web server on port 8080 where you can browse all transcript files. The viewer provides an interactive interface to browse transcripts from the run, view conversation flows with proper formatting, see judgment scores and justifications, and filter/search through transcripts.

Note: When using Weights & Biases, the pipeline automatically downloads transcript files to a local directory (configurable via EXTERNAL_TRANSCRIPTS_DIR in globals.py). You can then point the viewer to that directory (e.g., --dir /workspace/transcripts).