If my support system is small-scale, can I simplify by merging the three memory types together at first, then split them later once scale grows?
Merging them at small scale genuinely does save initial architectural complexity, but the key factor in deciding whether to split isn't "how many users you have" — it's "how high-risk the memory content is." If your support scenario doesn't involve refunds, compensation, or any action directly affecting fund flow at all, and is purely answering general inquiries, merging is relatively low-risk, and splitting later isn't a big problem. But if your support scenario handles refund promises from the start, even with a small user base, it's worth drawing the line between "general memory" and "memory involving fund flow" up front — don't let a small scale be a reason to simplify away this security consideration too.
A safer practical approach is that even if the technical implementation temporarily shares a single storage mechanism, you should still tag each memory entry's type and risk level clearly at the data-structure level from the start. That way, when scale grows enough to need splitting, you don't have to go back and reclassify a whole batch of historical data. Adding this tagging at the start is cheap; adding it retroactively later is much more expensive.
Is there a reasonably safe default approach in practice for the cross-contact memory-sharing question?
A reasonably safe default is "not shared by default, shared only with explicit authorization" — rather than the reverse assumption that contacts under the same enterprise account naturally get to see each other's interaction history. The reasoning behind this principle is that contacts under an enterprise account can vary widely in their actual scope of authority and trust level — whether a point of contact handling daily interactions and a temporary staff member who occasionally reaches out on someone's behalf should have the same access to historical records usually shouldn't be assumed by the system itself; it should be explicitly set by the enterprise account's administrator.
If your product needs to support "multiple contacts sharing the same interaction history," a more robust approach in practice is making this sharing relationship an independent setting the administrator can adjust at any time, rather than hardcoding it into the memory architecture's underlying logic. That way, when an enterprise customer needs to adjust access permissions (a contact leaves the company and access needs to be revoked immediately, say), it can simply be a settings change, without needing to touch the structure of the memory storage itself.
If an attacker already successfully wrote a fabricated refund promise into the memory store, is there any way for the support system to trace it back after the fact?
This is exactly why provenance tracking can't be an optional feature — it should be a required part of memory architecture. If every memory entry genuinely records its write source (which conversation, which account, what time), once a refund promise turns out to be questionable, you can trace it directly back to the specific interaction that generated that memory, review the conversation content from that time, and determine whether this was a genuine agreement between the user and a legitimate support agent or fabricated content that was injected. Without this layer of tracking, once memory gets poisoned, the usual response is casting suspicion across an entire batch and reviewing all of it — forensic cost runs far higher than with a tracking mechanism in place.
In practice, this can also pair with an additional mitigation: for memory entries involving fund flow, alongside recording the source, also record a status field for whether "this promise has been humanly reviewed." Any request that would trigger an actual refund action must first confirm this status field reads "reviewed" — rather than executing directly just because related content turns up in the memory store. This extra status field is essentially building an independent verification gate, separate from the memory content itself, directly into the memory architecture.
This whole design sounds like a substantial engineering investment — is there a more practical priority order for a team just starting out?
With limited resources, priority can be ordered by "degree of real risk involved." First priority is isolating memory that involves fund flow (refunds, compensation, billing adjustments), adding provenance tracking and secondary verification — it's fine if other memory types are temporarily merged, since this is the area where getting it wrong carries the most direct financial and reputational cost. Second priority is the isolation scope of user memory, ensuring no privacy-level cross-user data exposure — once this goes wrong, remediation cost and reputational damage are usually high, making it worth investing in early. Third priority is fine-grained division of memory type itself (storing and retrieving semantic, episodic, and procedural memory separately) — this mainly affects user experience quality and system maintainability, and can be optimized incrementally as scale grows without needing to be fully nailed in the first version.
The logic behind this priority order is putting limited engineering resources first into the area where getting it wrong is most costly, rather than distributing evenly across every design consideration. For most startup teams at an early stage, the more practical approach is making sure the high-risk piece is done correctly first, letting everything else run with a rougher implementation initially, and reinforcing incrementally based on issues actually observed.
"Add memory to our support agent" — this request sounds simple, but it actually hides at least three questions that need answering separately: how long to remember, what kind of content to remember, and how to prevent that memory from being poisoned once it exists. Most teams only work out the first question before starting implementation, jumping straight to picking a vector database, and the other two questions often only get addressed after something has already gone wrong.
A typical support agent task usually involves three different memory types at once, but not all are equally important. Semantic memory handles remembering general user information — account tier, preferred language, previously stated preferences. This kind of memory suits structured storage, since it's stable fact that doesn't much need to preserve the time context of "when it was learned." Episodic memory handles remembering specific past interactions — "this user's order was delayed last time, and support promised a coupon as compensation." This kind of memory needs to preserve chronological order and causality; without this layer, a user has to re-explain their background every single time, noticeably degrading experience — this is also the most easily underestimated, yet actually most critical, memory type in most support scenarios. Procedural memory handles remembering "the standard process for handling a given type of issue" — a refund request needs to first verify the order number, then confirm the refund policy applies, and only then trigger the refund action. This kind of memory usually doesn't need to vary by user; it's organizational-level knowledge shared across all support agents, and shouldn't be mixed into the same storage space as individual users' private memory.
A common mistake in practice is dumping all three memory types into the same vector database, handled by the same retrieval logic — this forces semantic similarity retrieval to take on a job it isn't built for. Procedural memory needs retrieval that precisely matches process steps; episodic memory needs retrieval that preserves chronological order. Forcing all of this through a mechanism that only does semantic similarity comparison can easily surface content that's semantically similar but actually inapplicable at a critical moment.
Another easily overlooked design decision in support scenarios is how to divide memory ownership scope. The most basic split is "private memory belonging to a single user" versus "organizational memory shared across users," but support system complexity usually needs one more layer of division: multiple contacts might exist under the same enterprise account — should these contacts share the same interaction history, or each have their own independent one? When a support agent hands off (transferring from tier-one to tier-two support), should memory transfer completely, or should sensitive information be filtered out first before handoff? There's no standard answer to these decisions — they need to be designed against actual business processes — but getting the division wrong usually has privacy consequences: information that shouldn't be shared gets mistakenly exposed to an unrelated user or support agent because the memory scope was designed poorly.
A support agent's long-term memory architecture is inherently a high-risk target for context poisoning attacks — an attacker only needs to use the normal channel of a support conversation to get the agent to write fabricated content into memory (disguised as a "previously approved refund promise," say). Any future interaction that happens to trigger retrieval of that memory entry could then be misled by the fake data the attacker planted. The core defense principle is establishing provenance tracking and trust partitioning for memory content: system policies and verified facts go into a read-only partition requiring human review for any change; ordinary memory generated through user conversation gets stored in a user-specific isolated partition, preventing cross-user poisoning spread. For support scenarios specifically, there's one additional detail worth watching: memory content that directly affects fund flow, like refund promises or compensation amounts — even if it's a genuinely legitimate memory generated from the user's own past interactions — deserves a secondary verification step independent of the memory itself before being cited to trigger an actual action, rather than simply trusting that "this was retrieved from the memory store, so it must be true."
If you're designing memory architecture for a support system, thinking through "which type to remember" and "how to protect it" as two separate questions directly reduces downstream maintenance cost and security risk. Confirming architecture first, then choosing the technical implementation, costs far less in rework than picking a vector database first and only discovering an architectural flaw afterward. And support scenarios are a relatively easy-to-reach attack surface for attackers, with memory content directly involving real fund flow like refund amounts — every gap in memory architecture design can ultimately translate directly into a concrete financial loss, a user being misled or a business being defrauded of a refund, not just technical debt.