Reviewing theme pull requests for twenty client stores: an agency setup

An agency that builds and maintains Shopify themes usually has one repository per client, each tied to a different store, each with its own merchant content, and a small team moving between them. A reviewer who opens a pull request on Monday morning may not have touched that client's theme in a month. They don't remember which alternate templates the merchant uses, which sections the agency's standards say need presets, or whether this theme is one the client wants submitted to the Theme Store.

This article is a playbook for running Auditor across that kind of portfolio: twenty client repositories is a typical number, but nothing here depends on it. It ends with a suggested rollout plan. With Black Friday ten days out, many agencies are entering a code freeze, and the first two weeks of that plan post nothing on pull requests, so they fit inside one.

One store per repository

Every Shopify feature in Auditor reads a store: theme previews deploy to it, the merchant content check reads its published theme, and theme documentation is written from it. Each repository has exactly one store, set at the top of its Shopify tab and used by everything below. For an agency that means each client repository points at that client's store, and a review of one client's pull request never reads another client's content.

There are two ways to reach a store:

• A Theme Access password per repository. The client installs Shopify's Theme Access app on their store and creates a password for you. It can only touch themes and isn't tied to a person, so it keeps working when someone leaves the agency.
• The team's Shopify sign-in, made once under Add-ons → Shopify on the Auditor page. Each repository can then search that account's stores. Deploys act as the account that signed in, and Shopify ends the session from time to time, so Auditor asks for a new sign-in.

The password is the steadier choice for many clients. The sign-in has one thing the password doesn't: a Theme Access password can't read metafield definitions, so the merchant content check's metafield part only runs through the sign-in.

Setting up twenty repositories in the console is twenty rounds of clicks. Since October 14, the Auditor CLI and the repositories API do it from a terminal. Install the GitHub App on the repositories first, which only the console can do. Then, with a key that has Auditor Permissions:

export IB_API_KEY=ib_live_…

auditor repos   # every repository the GitHub App can see, and what is on

# One line per client: asks for that store's Theme Access password, checks it
# against the store, stores it encrypted, turns on reviews and previews
auditor preview set --repo your-agency/client-one-theme \
  --store client-one.myshopify.com --password --enable-repo
auditor preview set --repo your-agency/client-two-theme \
  --store client-two.myshopify.com --password --enable-repo

Passwords are read from the environment or asked for, never taken as arguments, so they stay out of your shell history. A store behind its password page also needs --storefront-password, which Auditor uses only to load pages. auditor status --repo your-agency/client-one-theme shows one repository's setup and its last preview.

Agency standards as team rulesets

Most agencies have standards that hold across every client: how sections are built, what never goes in a theme, what a reviewer always checks. Written once as a team ruleset, they're checked on every client repository that matches.

Team rulesets are written on the Auditor page under Team rulesets, in the same Markdown format as a repository's .rules/ files: a title, severity and applies lines, a description, and optional ok and bad examples. Each rule's id gets a team/ prefix, so it never clashes with a client repository's own rules. A rule might read:

# New sections ship with a preset
severity: medium
applies: sections/*.liquid

A section added in this change has at least one entry under "presets" in its
schema. Without one, merchants can't add it from the theme editor.

Three settings make rulesets work across a portfolio:

• Applies to takes repository patterns, one per line, such as your-agency/*-theme. Empty means every repository Auditor reviews.
• Dry run checks the rules and records findings on the ruleset's page without posting them. Every new ruleset starts here, so you can read what a rule would say on twenty repositories before any developer sees it. Switch it to Enforced when the findings look right.
• Required means repositories in scope can't detach the ruleset, and neither .rules/ignore.md nor dismissing a finding turns its rules off. Only a team admin can change it. Use it for the few rules that are never negotiable. Optional rulesets can be detached per repository, or excepted for a path in that repository's ignore.md.

Client-specific conventions stay in each repository's own .rules/ folder, read from the base branch, so a pull request can't loosen the rules it's checked against. The deep review also reads AGENTS.md, CLAUDE.md and Cursor rules as context, which helps when a client's theme came from another agency with its own conventions. Developers can run the same checks before pushing with auditor review, and enforced team rulesets apply there too.

Previews for client sign-off

Agencies already send clients preview links. The difference with theme previews is that nobody has to make them. Each reviewed pull request gets its own unpublished theme on the client's store, updated on every push, and the link stays the same for the whole review. The merchant content the repository ignores is copied from the published theme, so the client sees their own homepage with the change in it, not a demo store.

That makes the pull request the sign-off record. Paste the preview link into the client thread, collect approval, merge. A render check on each deploy confirms the pages you list load without Liquid errors, so the client never opens a broken link. Previews are never published, and they're deleted when the pull request closes. A repository keeps five preview themes at once by default, and you can raise or lower that, since Shopify limits how many themes a store holds.

Checks that know the client's store

Two checks matter more on client work than on in-house themes, because the reviewer knows the client's store least.

The merchant content check reads the client's published theme for whatever a pull request removes: a section, a block type, a setting id, an alternate template. It comments with what would be lost, such as "12 products use product.bundle; removing it falls them back to product". It never writes to the store and calls no model, so it costs nothing.

Theme Check runs in Thorough reviews of a theme, with the repository's own .theme-check.yml, and posts offenses on the lines the pull request added. Depth is one of the settings a repository can override, so a large client theme can run Thorough while a small one runs Standard. Review depth covers what each one runs.

If the agency also builds themes for the Theme Store, turn on Theme Store readiness in those repositories' overrides. Each review then checks the requirements a theme's files can show, such as the required templates, app blocks in the product sections, and no Sass or minified files. For the requirements that need judgement, start a ruleset from the Theme Store template, and set its Applies to to those repositories only.

Spending across every client

Auditor has no seat price. Each review is billed as model requests to the team's Auditor key. Costs and limits says what each review is billed for. Across twenty repositories, the two caps under Spending on the Auditor page's Reviews tab keep the total predictable:

• Monthly cap: the most Auditor may spend in a UTC month across every repository, previews included. Once it's reached, new reviews are skipped with a neutral check titled Monthly spend cap reached, and reviews start again on the 1st. Billing contacts get budget alerts as it fills.
• Per pull request: the most Auditor may spend on one pull request across its reviews and fixes. It's checked before each review starts, so a review that starts under it runs to the end, and later pushes are skipped with a neutral check.

Both are team settings. If you bill clients for review time, the review cost is in each review's check run, and every request is in Logs. Skip rules keep bot pull requests out of the bill entirely: list dependabot/** or renovate[bot] under Skip pull requests, and those get no review, no check and no charge.

Recurring work across every client

Reviews cover what arrives as a pull request. Automations cover the rest: work that should happen in every client repository on a schedule or an event, written once for the team. Scope one to your-agency/*-theme and it runs in each matching repository, including ones you add later. Three are worth setting up on a portfolio:

• Shopify API deprecations, weekly, so the next API version isn't a surprise in twenty repositories at once.
• Triage failed Actions, on a failed check on a pull request, so a red build comes with its cause.
• Docs drift, on a push to the default branch, for clients whose handover notes live in the repository.

A client repository that doesn't fit can turn an automation off, or replace its instruction, from its settings drawer. Runs are billed like reviews, each with a per-run budget, and the monthly cap above applies on top. Start each one with Dry run, which posts nothing.

What clients see

Many clients read their own pull requests, and most want a guide at handover. Three features shape what they get.

The white label add-on, $49 a month from the team's balance, posts reviews from a GitHub App your organization owns, with your name, your picture and your own command word. A developer writes @your-word review, and @auditor review keeps working. The text on the pull request no longer names Inference Bureau, except the address of each comment's Configure link. Hide the cost of each review leaves the cost line out of the check run, and costs stay in Logs and Billing. The check run takes your App's name, so update any branch protection rule that requires the Auditor check.

The Viewer role lets a client read without touching. Invite them as a Viewer with no projects, then give them Auditor View on their own repositories, picked by name or by a rule such as your-agency/client-one-* that also covers repositories added later. They see those repositories' reviews and costs and change nothing. Members, Webhooks, Management keys, billing and the team balance are hidden from them. Auditor View includes each review's cost, so decide whether that's fine before you invite a client.

For your own developers, Manage on a set of repositories gives them those repositories' settings, rulesets and theme docs, while team-wide settings stay with owners and admins.

At handover, clients ask how to use the sections the agency built. Theme documentation writes that guide from the theme: a page for every section, theme block and group of theme settings, saying what it's for and what each option does, with pictures of each section on the client's own storefront. It's published at an address you choose, with an optional password, the client's logo and colors.

Your own notes go above or below any page, written in the dashboard or kept in the repository under .ib/, so a handover note lives next to the code it describes. It's off until you turn it on for a repository, on the same Shopify tab, and uses the store set there even when previews are off.

A suggested rollout

This is a plan we'd suggest, not a requirement. Adjust it to the size of your team.

Week 1: three repositories, dry run. Pick three active client repositories: one large, one small, one you know well. Set up their stores with auditor preview set. Write your agency standards as one team ruleset, apply it to those three, and leave it in dry run. Set the monthly cap to a figure you're comfortable losing to a mistake, and a per-pull-request cap.

Week 2: read the dry run. Open the ruleset's page and read every finding. Rewrite rules that fired on code you're fine with, and delete the ones nobody would act on. Compare the deep review's findings with what your reviewers caught. Set depth per repository.

Week 3: enforce, and widen. Switch the ruleset to enforced. Mark the two or three non-negotiable rules required, in a ruleset of their own. Set Applies to to every client theme and set up the remaining stores. Add skip rules for bots.

Week 4: clients. Turn on white label if you want reviews under your name. Send preview links for sign-off from the pull request. Invite the clients who want access as Viewers, and set up theme docs for the next handover.

If your freeze starts this week, weeks 1 and 2 fit inside it: dry-run findings post nothing, and the few hotfixes that go out still get the merchant content check. When the freeze lifts, run auditor drift on each client repository before the first deploy. Merchants edit their live themes heavily for campaigns, and a deploy from a branch that's behind the store can undo that work. Comment @auditor pull on a pull request after its preview deploys, and Auditor opens a pull request with the store's versions, so the client's campaign edits go through review like everything else.