We had an LLM turn plain English into crypto alert rules. Writing the rule was easy, keeping its numbers honest was the actual work

a couple of months ago i posted about wiring an llm to a hyperliquid account over mcp, and how much guardrail work the order side needed. someone on that post pointed out that a tool list bounds what the model can ask for, not what the server can sign. we ended up taking that to its conclusion and deleted the order tools outright. the mcp server is read only now. the llm moved to a job where a mistake costs a notification instead of a position. you describe what you want to catch in plain english, it turns that into an alert rule, backtests it on the last 1000 candles per coin and proposes it. writing the rule was the easy part again. these were the actual work. \*\*the model will quote numbers no tool returned.\*\* the backtest said a rule would send about 202 alerts a day. the reply said "202 without the cooldown, 26 to 28 a day with it". the 202 already included the cooldown and nothing had ever returned 26. the user said 26 a day was fine, created it, and got 9 alerts in the first 13 minutes. every per day figure in an answer is now checked against what the tools returned that turn, and an answer with a number from nowhere goes back to the model once. \*\*direction has to live inside the backtest, not on the label.\*\* someone asked for a short setup with positive edge on 4h. the backtest read every forward return as a long, so the model recommended the rule after which price rose the most (a 55.9% "win rate", which is 44% as a short) and called the best short on the tape "weak". direction is now a backtest input that flips the sign of returns, win rate, edge and excursions. \*\*count alerts, not episodes.\*\* "rsi below 70" looks harmless if you count how often it becomes true. the engine re-fires every cooldown for as long as it stays true, so on 1h that's about 6 alerts per coin per day. the backtest now scores every bar the engine would actually alert on. \*\*make the backtest a twin of the live engine, then diff them.\*\* same library, same defaults, same quirks. the diff found bugs in the live engine, not in the backtest. the indicator cache was keyed by symbol and params but not by bar, so the "previous" macd came back as the current one and a macd cross could never fire. and an ema 200 computed on exactly 200 candles is just its sma seed. latest run was 60 coins, 3 timeframes, every rule type: 22,139 cases, zero mismatches. \*\*wrappers fill gaps with plausible numbers.\*\* ours turned "not enough data" into defaults, so adx read 0 on brand new listings and "adx below 20" fired on all of them, and an rsi of exactly 0 became 50 through an \|| 50\. the technical indicators npm package counts a bar with an unchanged typical price as negative money flow (tradingview counts it as neither), which pulled mfi down by up to 7.8 points. missing data is null now and mfi is computed by hand. \*\*and the boring one, the model can't create anything.\*\* it proposes, the app renders a card, and only the user's tap creates the alert. it can't even print the card itself. an invented card id gets swapped for the real one, or the answer goes back. still wouldn't let an llm decide what to trade. as a way to turn "tell me when x happens" into a rule you can actually inspect, with honest numbers next to it, it's been better than i expected, as long as every number it says came from a tool. disclosure, i help build traderspy. it does alerts and research, no trading. the builder is at traderspy.app/strategies and the read only mcp connector at traderspy.app/mcp if anyone wants to poke at it, happy to go deeper on any of the above.