Skip to main content
Use Loop automations to investigate project data on a schedule and optionally send results to Slack or a webhook. For example:
  • Review cost or latency on a schedule.
  • Check whether quality is degrading.
  • Summarize recent errors.
  • Review logs using the filters from a dashboard.
  • Send a Slack or webhook digest after each run.
Enable Patterns to discover recurring problems and trends your scorers may miss, with supporting traces and suggested next steps. The discovery instruction and write permissions are preconfigured for you.

Create a Loop automation

1

Configure the automation

Go to Settings > Automations, click + Automation, and select Loop.Enter a Name to identify the automation in Settings and an optional Description, then configure:
  • Instruction: Describe what to inspect, how deeply to investigate, what to return, and when to stop. Use these techniques to make runs more consistent:
    • Pin the time range. Use the automation’s query window to keep consecutive runs comparable.
    • Specify the output. Define the format and columns so recurring digests are consistent.
    • Define exclusions. State what Loop should exclude or avoid inferring, such as guessing names or treating explicitly requested behavior as a failure.
    See Loop automation recipes for complete examples you can adapt.
  • Frequency: Choose Daily, Weekly, Interval, or Custom (cron). Daily and weekly schedules include a run time and show the next scheduled run. Custom cron uses five fields: minute, hour, day of month, month, and day of week. Select a timezone beside the cron expression. Schedule previews use the schedule’s timezone.
  • Agent configuration: Choose the model, reasoning effort, and whether to use Braintrust’s built-in models or your own AI providers. Built-in models use your monthly model credits. Other models bill to their providers. Cost depends on the model and how much data each run inspects, so keep instructions focused on a narrow slice of traces. See Models and providers and automatic pausing.
  • Default query range: Choose 3 hours, 6 hours, 12 hours, 1 day, 3 days, 7 days, or a custom range. Each run inspects the full range, not only traces created since the previous run.
2

Set write permissions

In an interactive thread, actions that change something pause for your approval. A scheduled run has nobody to ask, so it relies on its Write tool permissions instead.
  • Read-only tools never require approval, so a run can always inspect project data such as traces, existing patterns, experiments, and datasets.
  • Automations created from the Patterns page can create and update pattern records by default. A pattern record can include trace evidence, monitor chart configurations, and recommendations for measuring the pattern later. Creating scorers, facets, or other project objects requires the corresponding write tools.
  • To let a run create or modify other project objects, add the corresponding tools. The dropdown lists what each one permits.
Enabling a tool lets scheduled runs make that kind of change without asking. Choose only the tools the automation needs.
Allowing a run to create or update online scoring rules can start work that continues after the run ends. Online scoring rules score production traffic continuously and keep costing money until you stop them.
Objects a run creates are not linked back to any pattern it recorded. If a pattern’s monitor chart measures a scorer or facet the run created, deleting that object stops new data from reaching the chart.
3

Choose destinations

Destinations are optional. Every run’s results are available in Braintrust on the automation’s Past runs tab, and any patterns it records appear in the Patterns list. Add a destination when you also want each run pushed to your team.Click + Destination in the configuration panel, then choose Send to webhook or Send to Slack.
  • Slack. Select a channel to post a text digest. This requires a connected Slack workspace. Every message opens with the project name in bold on its own line. Use the destination’s Formatting prompt to tell Loop how to format the rest of the message.
  • Webhook. Enter a URL to send run results to an external service. Use the Formatting prompt to describe the JSON payload Loop should produce.
If no Slack workspace is connected, the destination menu depends on your organization permissions:
  • Owners or a custom permission group with Manage settings: Click Connect to Slack to connect a workspace in a new tab. When you return, the workspace list refreshes and the menu offers Send to Slack.
  • Without those permissions: The menu shows Ask an org admin to connect Slack.
Destinations belong to automation runs. Patterns that Loop records during an interactive chat appear in the Patterns list but do not trigger Slack or webhook destinations.
Click Create to save the automation. To run a saved automation immediately, open it and click Run now. Braintrust queues the run and shows a link to the Loop thread when it’s ready.

Loop automation recipes

Adapt these recipes to review cost and quality, summarize errors, or discover patterns. Each includes a schedule, query range, write permissions, and an instruction you can customize for your project.
Posts a cost breakdown to Slack every Monday morning.
Instruction
Compares existing quality scores with the previous day and investigates possible regressions. This example uses scores.factuality, with higher values indicating better quality. Replace it with a score your project already records and adjust the direction if lower values are better.
Instruction
Summarizes errors in the same subset of logs you monitor on a dashboard. Replace metadata.environment = 'production' with the filter conditions from your dashboard. Include the conditions in the instruction so the run has the scope it needs.
Instruction
Narrows pattern discovery to one concern, using an existing facet to choose which traces to investigate. Run this alongside the default Pattern discovery automation when a specific area deserves its own cadence.
Instruction

Review past runs

Each run creates a read-only thread recording the work Loop did. To find it, open the automation, select the Past runs tab, and select a run. These threads stay with their automation and don’t appear in the thread list on the Loop page. They also can’t be deleted the way an interactive thread can. To ask follow-up questions from a run, select Continue. This forks the read-only transcript into a private, editable thread, which is the fastest way to work out why a run behaved the way it did.

Pause or delete an automation

Pause an automation to stop its scheduled runs while keeping its configuration and history. Do this when you’re tuning an instruction, controlling spend, or investigating a run that behaved unexpectedly. When you’re ready, you can re-enable the automation. To pause, open the automation and turn off Active. The rule then shows as Paused in Settings > Automations. Turn Active back on to resume the schedule.
An automation configured to use built-in models does not switch to your own AI providers when its model credits are exhausted.An automation also shows as Paused automatically when Braintrust detects at read time that it cannot run. This happens when your model credits are depleted, the selected built-in model is unavailable, or built-in models are disabled for your organization. The automation list marks it with a warning indicator, and opening the automation shows a banner with the reason and a link to the relevant settings.
To remove an automation entirely, delete the rule from Settings > Automations.
Deleting an automation is permanent. If you might want the automation again, pause it instead.
Neither pausing nor deleting an automation affects the objects its runs already created. Patterns, scorers, facets, and other objects a run produced remain in your project, and any online scoring rules a run created keep running until you stop them separately.

Next steps