"Where is my order?" is one of the most common questions in a store's support inbox, and it is the easiest one to automate badly. A chat assistant that answers it has to read real order data: items, totals, carrier, tracking numbers, the city a parcel was last scanned in. If it hands that data to anyone who types a plausible order number, it has turned a convenience into a data leak. The policy questions next to it carry the opposite risk. A language model asked about a returns window that the store never wrote down will often make one up. While building DEV3 AI Support Chat & Order Tracking Assistant, DEV3's Shopify app for order tracking and store-policy answers, we treated both as security problems. This article covers how the app verifies who is asking before it shows an order, how it limits guessing, and how its retrieval layer says "I don't know" without asking the model at all. The app is in final QA before submission to the Shopify App Store; real-store testing is the next step.
The request path
The storefront widget never talks to the app directly. Every call goes through the Shopify App Proxy at /apps/concierge/*, which signs each request with an HMAC and adds the shop domain and, for a signed-in shopper, logged_in_customer_id. The app verifies the signature before it reads any parameter (condensed from app/lib/proxy.server.ts):
export async function proxyContext(request: Request): Promise<ProxyContext> {
await authenticate.public.appProxy(request); // throws 400/401 on a bad signature
const url = new URL(request.url);
const shop = url.searchParams.get("shop") ?? "";
if (!/^[a-z0-9][a-z0-9-]*\.myshopify\.com$/.test(shop)) throw json({ error: "bad_shop" }, 400);
const cid = url.searchParams.get("logged_in_customer_id") ?? "";
const ip = (request.headers.get("x-forwarded-for") ?? "").split(",")[0].trim() || request.headers.get("x-real-ip") || "unknown";
return { shop, customerId: /^\d{1,20}$/.test(cid) ? cid : null, ipKey: hashIp(ip) };
}
Because Shopify signs the customer id, the app can trust it: a shopper can't become customer 42 by editing a query string. The client IP is used only as a salted hash for rate-limit keys.
From there, handleChat in app/lib/chat.server.ts runs a fixed sequence: rate limits, conversation lookup with the plan's monthly limit, intent and sentiment classification, then routing. Order tracking goes to a verified lookup, policy and product questions to retrieval-augmented answers, and disputes or repeated misses to a hand-off. Every dependency is injected, so the whole flow runs in unit tests with mocked Shopify and AI calls.
Two ways to prove an order is yours
The header comment of app/lib/orders.server.ts states the rule:
Nothing about an order is revealed unless either the storefront session is a signed-in customer (App Proxy
logged_in_customer_id) who owns it, or the customer supplied the order number AND the order's email, and both match. Mismatches and unknown orders get the same generic answer; attempts are rate-limited.
The router tries three paths in order:
- Orders already verified in this conversation. Their Shopify IDs are kept on the conversation record, so "and when will it arrive?" doesn't ask for the email again.
- A signed-in customer. The app queries that customer's five most recent orders. If there is one, it shows it. If there are several, it shows a list to pick from. An order number that isn't on the account is not trusted just because the shopper is logged in. It falls through to the guest check.
- A guest. The order number and checkout email must both be present. If either is missing, the widget shows a small form instead of guessing.
The guest check
The guest path is where an attacker would aim, so it is the most defended (condensed from verifyGuestOrder):
const orderNumber = cleanOrderNumber(args.orderNumber);
const email = normalizeEmail(args.email);
if (!orderNumber || !email) return { ok: false, reason: "invalid" };
// Brute-force protection: per session, per IP and per order number (across sessions).
const limits = await Promise.all([
rateLimit(args.redis, `concierge:lookup:s:${args.shop}:${args.sessionId}`, 5, 15 * 60),
rateLimit(args.redis, `concierge:lookup:ip:${args.shop}:${args.ipKey}`, 15, 60 * 60),
rateLimit(args.redis, `concierge:lookup:o:${args.shop}:${canonicalOrder(orderNumber)}`, 8, 60 * 60),
]);
if (limits.some((l) => !l.ok)) return { ok: false, reason: "rate_limited" };
const data = await gql(args.admin, ORDER_BY_NAME_QUERY, { q: orderSearchQuery(orderNumber) });
const wanted = canonicalOrder(orderNumber);
const candidates = (data?.orders?.nodes ?? []).filter((o) => canonicalOrder(o.name ?? "") === wanted || String(o.number) === wanted);
let match: any = null;
for (const o of candidates) {
// Evaluate both comparisons for every candidate (no early exit on the first field).
const a = constantTimeEqual(normalizeEmail(o.email), email);
const b = constantTimeEqual(normalizeEmail(o.customer?.defaultEmailAddress?.emailAddress), email);
if ((a || b) && !match) match = o;
}
return match ? { ok: true, order: match } : { ok: false, reason: "not_found" };
Several decisions in this function work together.
Input is validated before Shopify sees it. cleanOrderNumber accepts only ^#?[A-Za-z0-9][A-Za-z0-9_-]{0,30}$. The order number ends up inside an Admin API search string (name:#1001 OR name:1001), so anything that could change the query's meaning is rejected first. The test rejects invalid input without calling Shopify passes 1001) OR (name:* and asserts that admin.graphql was never called.
Search results are re-checked exactly. Shopify's order search is a search, not a key lookup. The code compares the canonical order name (upper case, no leading #) or the order number against what the customer typed, and the test does not accept a partial order-number match from the search confirms that #10011 is not accepted for 1001.
Emails are compared in constant time. Both sides are hashed and compared with Node's timingSafeEqual:
export function constantTimeEqual(a: string, b: string): boolean {
const da = createHash("sha256").update(a).digest();
const db = createHash("sha256").update(b).digest();
return timingSafeEqual(da, db) && a.length === b.length;
}
Hashing first gives timingSafeEqual the equal-length buffers it requires, so timing doesn't reveal how many leading characters match. The loop also checks both emails for every candidate instead of stopping early.
Failure is uniform. An unknown order and a known order with the wrong email both return { ok: false, reason: "not_found" }, and the chat shows the same message for each: "I couldn't find an order matching that order number and email." A dedicated test asserts that the two results are identical, because if they differed, the chat would tell an attacker which order numbers exist.
Three rate limits, each for a different attacker
One limit is not enough because attackers vary different inputs:
- Per browser session, 5 per 15 minutes, which stops one visitor guessing emails for one order.
- Per IP hash, 15 per hour, which stops one client that keeps rotating sessions.
- Per order number across all sessions, 8 per hour, which stops many clients targeting one order.
The rateLimit helper, part of a shared core used across DEV3's apps, is a fixed-window Redis counter with an in-memory fallback. The rate-limits lookup attempts per session test sends seven guesses and asserts that the first five return not_found and the sixth returns rate_limited.
The chat endpoint has its own limits on top of these: 30 messages a minute per IP, 12 a minute per session and 200 a day per session.
Sessions can't be inherited
A shared computer is a quieter leak. If a signed-in customer tracks an order, signs out, and someone else opens the chat in the same tab, the widget still holds the same random session id. handleChat refuses to continue a conversation bound to a different customer id:
if (conv?.customerId && conv.customerId !== req.customerId) {
// Signed-out or different customer on the same browser tab: never show the previous customer's data.
throw new ChatError(409, "session_reset", "Your session changed. Starting a new conversation.");
}
The history endpoint applies the same check, the widget answers a 409 with a fresh session, and a test covers the case.
What the model sees, and what gets stored
Once an order is verified, the model gets a short plain-text summary built by orderFacts: status, payment and fulfilment state, items, carrier, tracking number, estimated delivery and the latest scan. Addresses and payment details are never part of it. The customer's question is wrapped in <untrusted> tags with a rule to ignore instructions inside them. The model writes one or two sentences. Items, tracking links and scan history are rendered from the structured OrderCard, so a hallucinated date can't change the card, and if the AI call fails a deterministic sentence takes its place.
Messages created by the lookup and hand-off forms store the email masked (j***@example.com), and full order objects are fetched live, not persisted.
RAG that says "I don't know"
The second half of the problem is answering from the store's policies and pages without inventing any. The knowledge base lives in Postgres with pgvector. Shopify policies, published pages and merchant-added URLs on the shop's own domain are chunked to about 800 tokens with a 100-token overlap, then embedded with gemini-embedding-001 at 768 dimensions. The truncated embeddings are L2-normalised so similarity scores are comparable.
An index Prisma won't fight
Prisma can't model an HNSW index. A plain index on the vector column would show up as drift on every migrate dev. The init migration therefore creates an expression index that Prisma ignores, on a half-precision cast, which also halves the index's memory:
CREATE INDEX "KnowledgeChunk_embedding_hnsw_idx" ON "KnowledgeChunk"
USING hnsw ((("embedding")::halfvec(768)) halfvec_cosine_ops);
The trade-off is that every query has to order by exactly the same expression, or Postgres won't use the index. searchChunks does, and it sets hnsw.iterative_scan = relaxed_order so that filtering by shop and enabled sources still returns k rows (condensed):
await tx.$executeRawUnsafe(`SET LOCAL hnsw.iterative_scan = relaxed_order`);
await tx.$executeRawUnsafe(`SET LOCAL hnsw.ef_search = 100`);
return tx.$queryRaw`
SELECT c."id", c."sourceId", s."title", s."url", c."content",
1 - (("embedding"::halfvec(768)) <=> (${vec}::vector(768))::halfvec(768)) AS similarity
FROM "KnowledgeChunk" c JOIN "KnowledgeSource" s ON s."id" = c."sourceId"
WHERE c."shop" = ${shop} AND s."enabled" = true AND c."embedding" IS NOT NULL
ORDER BY ("embedding"::halfvec(768)) <=> (${vec}::vector(768))::halfvec(768)
LIMIT ${k}`;
A test runs EXPLAIN with sequential scans disabled and asserts that the plan names KnowledgeChunk_embedding_hnsw_idx, so a future edit that breaks the expression match fails CI instead of silently falling back to a slow scan.
Two refusal gates
answerFromKnowledge can decline at two points. The first gate is a similarity threshold, and it runs before any generation call:
const all = await searchChunks(args.db, args.shop, vector, args.k ?? 6);
const hits = all.filter((h) => h.similarity >= args.minSimilarity);
if (!hits.length) return { answerable: false, answer: "", citations: [], hits: all };
The default minSimilarity of 0.60 came from measurement. In the live test run, a question about sending back a shirt ranked the refund policy first at 0.701, while unrelated chunks scored about 0.58–0.60. When nothing clears the threshold, the customer gets "I'm sorry, I don't know the answer to that from our store information" and an offer to reach the team. No tokens are spent, and the model has nothing to embellish.
The second gate covers weakly related passages that clear the threshold. Generation uses structured output with a schema of answerable, answer and source_ids, and the system prompt says to answer "ONLY with facts stated in the numbered store sources" and to set answerable=false otherwise. The live test checks this directly. It asks about gift wrapping with a handwritten card, drops the threshold to 0.4 so some passages get through, and asserts the model still returns answerable: false.
When the model does answer, citations come from the source_ids it reports, validated as integers within range and de-duplicated by source. Only https URLs are linked. In the live run, the refund-policy question was answered with "30 days" and cited the refund policy.
Misses lead to a person
Saying "I don't know" is only useful if it leads somewhere. Each miss increments failureCount. At the merchant's threshold (two by default), the assistant hands off instead: it asks for a reply-to email if needed, then creates a ticket with the transcript by email, in Zendesk or in Gorgias, with failed deliveries retried by a BullMQ worker. The says it doesn't know instead of guessing, and hands off after repeated misses test runs two nonsense product questions through the full flow and asserts the hand-off form appears on the second.
Trade-offs
- Guests must type their email. This adds friction compared with an order number alone. We accepted it because the order number on its own is a short, guessable sequence.
- Fixed-window limits allow a burst at the window boundary. That's acceptable when the per-order and per-IP limits are hourly.
- Orders older than 60 days need Shopify's
read_all_ordersscope. Without it, older lookups get the same generic "couldn't find" answer, which is safe but can confuse a returning customer. - The similarity threshold is a tuned number. It is configurable per shop (0.3–0.95), and the model-side refusal is the backstop when it's set too low.
Verification
The app's suite has 46 Vitest tests against real pgvector, mocked Admin GraphQL and helpdesks, and a real SMTP sink. Four live tests call the Google Gemini API for embeddings, grounded answers, refusals and intent classification, and typecheck is clean. GDPR customers/data_request, customers/redact and shop/redact handlers are implemented and tested. The app is in final QA before submission to the Shopify App Store; real-store testing is the next step. The case study covers the wider architecture: the Shadow DOM widget, hand-off adapters and merchant dashboard.
