Ren, our first model.
Public weights, public limits.
A decision model takes a situation and a list of options, then returns one option with a probability. To trust software to act alone, you need proof of when it's safe.
Model details
Ren is trained once. It can work from option names alone and gets better with a few examples. It's English only.
- Parameters
- 396M
- Training rows
- 422,906
- Training sets
- 38
- Compute
- About $129
- Weights licence
- CC BY-SA 4.0
- Version
- v1.1.0
Built with Muse Spark Open. Ren is independently developed for Meta's platform and is not an official Meta product.
Results
| Test | Ren | Comparison |
|---|---|---|
| Typed decisions, mixed test | 79.7% | OpenDecider-nano 79.6%, OpenDecider's run of Jev 1.13 75.4% |
| BANKING77, no examples | 69.0% | Jev 79.7%, from a published study |
| BANKING77, with examples | 89.9% with 8 | Jev 92.4% with 24, from the same study |
| MTOP, no examples | 76.0% | |
| MTOP, 8 examples | 92.4% |
Acting alone, at our 99% gate
- MTOP: Ren answered 21.0% alone and made 1 error in 923 answers. Its lower confidence bound is 99.40%.
- BANKING77: Ren answered 0% alone, because it couldn't prove it was safe. That gap is the research problem we're working on.
Speed and cost
- GPU: 26 ms, not counting the network.
- Through the endpoint: 424 ms median from India to one GPU in the US.
- CPU: 2.18 s.
- Cost: $0.083 per 1,000 decisions at full load. Jev works out to about $0.008.
Limits
- The result covers one seed with full history, and it doesn't certify a new business.
- When the right option was missing, Ren wrongly accepted 2 in 500, above its target of 1.
- No real business has run Ren yet.
- With no examples on BANKING77, Ren doesn't lead: Jev scores 79.7%.
What's next
- Build the GitHub connector.
- Test Ren against Jev on real decisions.
- Close the BANKING77 gap, so Ren can prove more of its answers are safe.