Explainable Semantic TM (xTM)

Explainable Semantic TM (xTM) is a retrieval‑first, terminology‑aware, explainable translation memory. It uses semantic search (FAISS + multilingual E5 embeddings) to find the best existing, approved translation by meaning — and shows why it was chosen. It is not a generative MT system; when confidence is low, it prefers to return [No translation].

The UI offers Basic (semantic k‑NN), Advanced (multi‑factor scoring with a “Why” panel), Hybrid (dense + BM25 fusion), and Compare All (fuzzy vs semantic) modes.

• Is: a semantic “translation memory by meaning” that retrieves trusted translations from your bilingual corpus.
• Isn’t: a generative translator — no text is invented; guardrails return “[No translation]” when uncertain.
• Why different: goes beyond fuzzy character matching; catches paraphrases and synonyms using embeddings.

1. Preprocess: normalize text for stable embeddings.
2. Embed: multilingual‑E5 with retrieval prefixes → 768‑dim vectors.
3. Index: build FAISS (IndexFlatIP) over source embeddings.
4. Retrieve: k‑NN returns nearest source sentences and their targets.
5. Re‑rank: combine factors — semantic similarity, domain, length, context, terminology (glossary bonuses/penalties with strictness). Optional: hybrid BM25 fusion and cross‑encoder reranker when enabled.
6. Guardrails: if final score < threshold → return “[No translation]”.

• Repetitive or regulated content with strict terminology (technology, legal, medical, support).
• Teams with a bilingual memory who want retrieval by meaning, not string fuzziness.
• Workflows requiring explainability and guardrails (Why panel + safe “no translation”).

• Needs a bilingual corpus; won’t produce novel translations outside your memory.
• Quality depends on embedding model and data coverage; smaller memories reduce recall.
• Intentionally non‑generative — prioritizes safety and consistency over creativity.

• Reranking: optional cross‑encoder to tighten Top‑1 precision on larger k.
• Generative fallback: opt‑in LLM only when retrieval confidence is low (keep auditability).
• Stronger embeddings: evaluate BGE‑M3 or similar for higher accuracy.
• Online updates and CAT/TMX integrations.

• AI/ML: strong — embeddings, vector search, multi‑factor scoring.
• LLM: not used — non‑generative by design.
• MT: complementary — semantic TM retrieval instead of neural MT.
• RAG: strong “R” — a high‑quality retriever that can feed a generator if desired.

• Use the Featured Demos and click “Run Compare” to see fuzzy vs semantic vs advanced.
• Switch to Advanced, select a Domain, and adjust Terminology strictness to see factor effects.
• Try Hybrid mode to blend dense semantics with lexical BM25 for codes/IDs/keywords.
• “autodetection” uses no domain filter; set a domain to see disambiguation (e.g., python/bank).

Sample outcomes on the curated demo set:

• Translation speed: ~30–40ms incl. embedding.
• Accuracy: 90–98% on PoC similarity clusters; 80%+ for domain queries.
• Throughput: ~25–30 translations/sec; Memory: ~500MB for ~300 pairs.

For details, see repository docs and scripts (e.g., scripts/xtm_metrics.py).