How does ScenarioTrace decide what broke in a Make.com scenario?

What does the analyzer actually look at?

It reads two run logs you paste from Make.com’s Module Inspector: one from a run that worked and one from a run that failed. It lines up the two runs module by module (by each module’s id) and compares each pair’s input, output, and error. Everything is a comparison — the working run is the baseline that tells the checks what “normal” looked like.

The deterministic checks run first and always. Most diagnoses finish here, with no AI involved. Each finding is ranked by how close it is to the root cause, and the closest one becomes the “most likely cause.” A set of checks called the silent checks only run when neither run showed an error and the HTTP status code did not change — that is the “it said success but the data is wrong” situation.

When does it use AI — and when does it not?

AI is the exception, not the norm. The deterministic engine runs first on every request. An AI model (Claude) is only called for one specific hard case: when a text answer collapsed to under 30% of the length it had in the working run — a quality-degradation case the plain checks can flag but not fully explain. The large majority of results never touch an AI model.

If the AI call times out or errors, the result falls back to the deterministic answer. And when no check matches at all, the analyzer returns an honest “no result” with guidance — it does not invent a cause. (A separate always-on AI pass for paid accounts is designed but not live.)

Which outright errors does it catch?

These read the failing run’s error text (most need only the failed run). They are the highest-confidence findings.

• Expired login / connection — the module’s login or API key expired, so it can no longer authenticate. Matched from error wording like 401/403, “token”, “expired”, “unauthorized”.
• Permission or scope block — the request was blocked by a missing permission, a narrowed scope, or an IP allowlist — not an expired login. Kept separate because “re-authorize” is the wrong fix here.
• Rate limit — too many requests too fast. Fires on HTTP status exactly 429, or wording like “rate limit”, “too many requests”, “quota”, “throttled”.
• Timeout — the request took too long or the connection dropped. Fires on status exactly 504 or timeout/socket wording (deliberately narrow — 500/502/503 are treated as a response change, not a timeout).
• Known Make error class — the error matches one of Make’s own named failure types (ConnectionError, DuplicateDataError, IncompleteDataError, DataError, BundleValidationError, InvalidConfigurationError, RuntimeError, or “missing value of required parameter”).
• Response or status changed — the server answered differently than before — either it returned data in the working run but now returns nothing plus an error, or its HTTP status code changed. A bare “status” field only counts as an HTTP code when both values are integers 100–599 (a word like “done→completed” is handled elsewhere).

How does it catch wrong or missing values?

These compare the two runs’ inputs and outputs field by field.

• A value changed — a field holds a different value than it did in the working run. If both values are URLs, it compares the query parameters one by one — a changed URL parameter is treated as a likely root cause.
• A field went empty — a field that had data is now null, undefined, or an empty string — usually an upstream module stopped feeding it.
• A field changed type — a field switched shape (text↔number, object↔array), which breaks modules expecting the old format.
• Wrong record returned — the response has the same shape but the id-like fields differ — you’re getting a different record’s data (matched on keys like id, uuid, sku, slug, email, username).
• Some fields went null — specific fields that had values now come back empty with no error — often a provider privacy or schema change.
• Placeholder text written in — a field now literally contains “undefined”, “null”, “NaN”, “[object Object]”, or an unrendered {{…}} template — an unmistakable broken mapping.
• Text was cut off — a text value is an exact prefix of the working value, kept to under half its length (and the original was at least 20 characters) — the signature of a size or encoding limit.
• Stale cached data — a field was empty in the input but its output matches the working run’s value exactly — Make likely reused previous-run data for an empty field.

How does it catch silent count problems?

“It said success, but the number of records is wrong.” These run only when neither run errored.

• Empty result — a list that had records now returns zero — a filter, search, or date range almost certainly stopped matching.
• Count way off — an output list came back with a very different number of items — it flags when the count is less than half, or more than double, the working run (a core “data” list flags on any change).
• A filter compared as text — with effectively identical inputs, an output list changed size by 10× or more — the sign of a filter comparing a date or number as plain text.
• Only the first page — the response has a “next page / cursor / total” indicator but only the first page of records was processed — you’re silently missing later pages.
• Duplicate records — an output list now repeats records the working run had only once, and the list grew — usually a trigger fired twice or a batch was re-processed (needs at least two repeated ids and a larger list).

How does it catch a changed response shape?

The upstream API changed the structure of what it sends back, silently breaking mappings that read the old shape. These also run only in the “no error” case.

• Fields disappeared — the response is missing fields it included before.
• A wrapper was removed — a wrapper object was dropped and its fields moved up to the top level, so code reading “wrapper.field” breaks.
• Fields were renamed — some fields were removed and others added; it pairs them up as likely renames (using a name-similarity threshold) and reports the rest as removals.
• List items changed shape — the list still arrives, but the objects inside it were renamed or lost fields.
• A status word changed — a status-like field’s wording changed (e.g. “pending→processing”). This one is hedged: the logs alone cannot tell a real break from a record legitimately moving to a new state, so it is reported at Medium confidence at most.
• A date format changed — a date field’s format changed (e.g. a numeric timestamp became an ISO date). Also hedged to Medium — a format change can break date parsing even when it’s the same instant, but it can also just be a new value.

How does it catch a module that didn't run?

• A filter or router blocked the chain — a module that ran in the working log did not run in the failing log, and no module reported an error — so a filter or router condition most likely stopped matching, and nothing was passed forward.
• Blocked by an earlier failure — a module didn’t run because an upstream module errored — reported as a downstream chain effect.

How sure is a result?

Every diagnosis shows a confidence level, set by how close the top finding is to the root cause:

• High — the strongest finding is a clear, close-to-root-cause signal (an explicit error, a changed input value, or a definite silent-data break) — or several findings agree.
• Medium — one or two weaker findings, including the two deliberately-hedged checks above (a status word or a date format that changed).
• Low / no result — nothing matched confidently. Rather than guess, the analyzer returns an honest “no result” with guidance on what to check or paste next.

What can it NOT detect?

Being honest about the edges is the point — this is a focused tool, not a catch-all.

• It needs two runs — a working run and a failing run. Without a working baseline to compare against, only the error-text checks (login, permission, rate limit, timeout, Make error class) could possibly fire.
• It reads run logs, not your design — it sees the per-module input, output, and error captured in a run. It cannot see the scenario’s blueprint — the filter and router conditions, the mappings, iterator or aggregator settings, or error-handler configuration. It can infer “a filter probably stopped matching,” but it cannot read the filter itself.
• Modules are matched by id — if the two pasted runs don’t line up (modules renumbered or re-added), pairs won’t match and the field-by-field checks can’t run.
• The silent checks need a clean run — the structure- and data-drift checks only run when neither run errored and the status code didn’t change. If the failing run has any error, those are skipped in favour of the error itself.
• Fixed thresholds miss quiet cases by design — to avoid false alarms, small changes are ignored on purpose — e.g. a record count that moved between half and double isn’t flagged, a list has to swing 10× for the “compared as text” check, and text has to be cut to under half to count as truncated.
• Two checks are intentionally uncertain — the “status word” and “date format” checks can’t tell a real break from a legitimate change, so they never report High.
• Genuinely new failures are reported as unknown — if a failure matches no known pattern, it says so — it does not fabricate a cause.

A note on privacy: the runs you paste are used only to produce your diagnosis in that request. See the privacy page for the details.