Atomic procedural memory + durable episodic extraction (dedup simplification, API renames, hardening)

[Aayush Kataria (aayush3011)]

Summary

Makes procedural memory a first-class, atomic skill/policy library and to run episodic extraction on the Durable Functions backend via the Cosmos DB change feed. Along the way it simplifies fact dedup, renames the public write/delete API, and hardens the durable cadence and procedural-activation paths.

Features

• Procedural memory- ProceduralRecord is now an atomic procedure (behavioral policy / workflow / decision rule / tool-usage / recovery strategy) with scope, activation conditions, steps, status, and source provenance. New retrieve_procedures() (context-aware semantic retrieval) and build_procedural_context(user_id, task=...) (deterministic compilation of the active, in-scope procedures into a system prompt). Provenance gating
  auto-activates only trusted and grounded sources; episode-distilled, document, inferred, or ungrounded procedures stay candidate and are excluded from the compiled prompt.
• Durable episodic extraction - the Functions backend segments episodes from the change feed on a per-thread cadence (EPISODE_EVAL_EVERY_N, plus EPISODE_IDLE_GAP_SECONDS, EPISODE_TOPIC_DRIFT, EPISODE_MAX_TURNS, EPISODE_MIN_TURNS), surfaced through the SDK, Bicep/azd, and .env.template.
• Delete helpers - delete_turn(), delete_thread_summary(), delete_user_summary(), and bulk delete_thread().

Breaking changes

• add_cosmos() → upsert_memory(); delete_cosmos() → delete_memory() (both clients + store).
• Procedural memory reshaped from a single compiled-prompt document to atomic procedures compiled on demand; old single-prompt procedural docs are not migrated.
• Write-time vector dedup removed (DEDUP_VECTOR_ENABLED + similarity knobs gone); fact dedup is now in-batch hash + deterministic-id create/409. Contradiction reconciliation is unchanged.

Fixes & hardening

• Durable cadence integrity (headline fix): episodic segmentation no longer stamps turns (episode_extracted_at). Stamping re-entered the turns change feed and mis-counted the per-thread cadence counter - first over-counting, then (after a naive skip) permanently under-counting and dropping facts under a lagging feed. Replaced with a per-thread episode_cursor doc ((created_at, id) watermark in the memories container) that never writes to the turns container, so each turn is counted exactly once. Also closes a partial-stamp duplicate-episode window.
• Monotonic watermarks - extract-count via a conditional patch, episode cursor via a read-check, so out-of-order concurrent orchestrations can't regress them.
• Procedural correctness - seed utility_score from the LLM's confidence (was discarded / hard-wired to 0.5); force candidate for ungrounded or episode-only self-labeled instructions (no auto-activation without real grounding); order source selection by created_at (not salience) so it can't silently drop sources missing salience.
• Sync/aio parity fixes in episodic extraction; removed dead code (vector-dedup helpers, inplace_updated telemetry) and never-shipped legacy-episodic back-compat.

Cleanups

• Dropped the unused (salience, created_at, id) composite index (code + Bicep).
• Fact hash-dedup no longer preloads existing hashes per extraction (in-batch + create/409 only), cutting per-turn latency.
• Default fact-extraction cadence is now every 2 turns (FACT_EXTRACTION_EVERY_N=2) across the SDK, .env.template, and the Functions deploy default.

Testing

• 1215 unit tests pass; ruff + format clean; integration suite collects (25 tests).

[Copilot]

Pull request overview

This PR makes procedural memory an atomic, retrievable policy/skill library and moves episodic extraction for the Durable Functions backend to a Cosmos DB change-feed–driven orchestration, while simplifying deduplication and renaming the public write/delete APIs.

Changes:

• Replace legacy vector dedup paths with in-batch exact dedup + deterministic-id create/409, and remove related thresholds/helpers.
• Introduce atomic ProceduralRecord procedures plus retrieve_procedures() and task-aware build_procedural_context(...).
• Add Durable episodic extraction orchestration/cadence controls and update SDK/docs/samples for upsert_memory() / delete_memory() API renames.

Reviewed changes

Copilot reviewed 100 out of 100 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_utils.py	Remove vector-similarity helper tests
tests/unit/test_thresholds.py	Update defaults; remove dedup env tests
tests/unit/test_reconcile.py	Update extract dedup expectations
tests/unit/test_process_now.py	Update API rename in test text
tests/unit/test_pipeline_confidence.py	Remove legacy dedup pinning
tests/unit/test_models.py	Expand procedural model coverage
tests/unit/test_cosmos_memory_client.py	Rename API tests; add delete helpers tests
tests/unit/test_auto_trigger.py	Adjust docstring for new defaults
tests/unit/store/test_memory_store.py	Rename store upsert test
tests/unit/services/test_procedural_retrieval.py	Add retrieve_procedures unit tests
tests/unit/services/test_pipeline_service.py	Adjust fakes for create/409 + procedures
tests/unit/services/test_persist_extracted.py	Rename add->upsert in fakes
tests/unit/services/test_mark_turns_extracted.py	New tests for extracted_at marking
tests/unit/services/test_extract_episodes.py	Adjust for episode_cursor docs
tests/unit/services/test_extract_dry.py	Update episodic cursor filtering logic
tests/unit/services/test_episodic_retrieval.py	Add warning tests for episodic opt-in
tests/unit/services/test_episode_boundary.py	Switch from per-turn stamps to cursor
tests/unit/services/test_chaos_extract_persist.py	Rename add->upsert in chaos fakes
tests/unit/processors/test_durable.py	Durable episodic extraction expectations updated
tests/unit/function_app/test_synthesize_procedural_orchestrator.py	Slim payload now reports procedures_created
tests/unit/function_app/test_orchestrators.py	Remove em_Dedup from orchestration chain
tests/unit/function_app/test_extract_episodes_orchestrator.py	New episodic extraction orchestrator tests
tests/unit/function_app/test_config_episode.py	New EPISODE_* config getter tests
tests/unit/function_app/test_change_feed.py	Add EPISODE_EVAL_EVERY_N triggering + monotonic watermark tests
tests/unit/aio/test_reconcile_telemetry.py	Remove async legacy dedup pinning
tests/unit/aio/test_process_now.py	Update API rename in async test text
tests/unit/aio/test_cosmos_memory_client.py	Rename async API tests; add async delete helpers
tests/unit/aio/store/test_memory_store.py	Rename async store upsert test
tests/unit/aio/services/test_procedural_retrieval_async.py	Async retrieve_procedures tests
tests/unit/aio/services/test_mark_turns_extracted_async.py	Async extracted_at marking tests
tests/unit/aio/services/test_extract_episodes_async.py	Async episode_cursor behavior assertions
tests/unit/aio/services/test_episodic_retrieval_async.py	Async episodic opt-in warning tests
tests/unit/aio/services/test_episode_boundary_async.py	Async cursor-based episodic boundaries
tests/unit/aio/processors/test_durable.py	Async durable episodic extraction behavior updated
tests/integration/test_ttl_lifecycle.py	Rename delete API usage
tests/integration/test_procedural_pipeline.py	New live procedural synthesis/retrieval/context test
tests/integration/test_full_pipeline.py	Rename APIs; remove vector-floor dedup integration test
tests/integration/test_episodic_pipeline.py	Rename API usage
tests/integration/test_async_full_pipeline.py	Rename APIs; remove async vector-floor dedup test
Samples/Scenarios/scenario_tagging_and_filtering.py	Rename add_cosmos usage
Samples/Scenarios/scenario_remote_processor.py	Rename add_cosmos usage
Samples/Scenarios/scenario_remote_processor_async.py	Rename add_cosmos usage
Samples/Scenarios/scenario_rag_with_memory.py	Rename CRUD API usage
Samples/Scenarios/scenario_multi_agent.py	Rename add_cosmos references
Samples/Scenarios/scenario_memory_reconciliation.py	Rename CRUD API usage
Samples/Scenarios/scenario_customer_support.py	Rename add_cosmos usage
Samples/Scenarios/scenario_counter_tuning.py	Rename add_cosmos usage
Samples/Scenarios/scenario_chat_memory.py	Rename APIs + sample text
Samples/Quickstarts/quickstart_cosmos.py	Rename CRUD API usage
Samples/Processing/processing_user_profile.py	Rename add_cosmos usage
Samples/Processing/processing_thread_summary.py	Rename add_cosmos usage
Samples/Processing/processing_fact_extraction.py	Rename add_cosmos usage
Samples/Processing/processing_episodic_memory.py	Rename delete API usage
Samples/Notebooks/Demo.ipynb	Rename add/delete APIs in notebook
Samples/Notebooks/Demo_function_app.ipynb	Rename add_cosmos references
Samples/Notebooks/Demo_function_app_async.ipynb	Rename add_cosmos references
Samples/Notebooks/Demo_async.ipynb	Rename add/delete APIs in async notebook
Samples/Advanced/advanced_search_patterns.py	Rename add_cosmos usage
Samples/Advanced/advanced_memory_lifecycle.py	Rename add/delete APIs
README.md	Update examples + default cadence docs
infra/README.md	Update default fact cadence
infra/modules/functions.bicep	Add EPISODE_* params/app settings
infra/modules/cosmos.bicep	Remove unused composite index
infra/main.parameters.json	Add EPISODE_* defaults; update cadence
infra/main.bicep	Add EPISODE_* params; update cadence docs
function_app/triggers/change_feed.py	Start ExtractEpisodesOrchestrator; add n_episode gating
function_app/shared/counters.py	Monotonic extract watermark via conditional patch
function_app/shared/config.py	Add EPISODE_* env parsing + float validation
function_app/orchestrators/synthesize_procedural.py	Slim payload now reports procedures_created
function_app/orchestrators/extract_memories.py	Remove Dedup activity; update docs/comments
function_app/orchestrators/extract_episodes.py	New episodic extraction orchestrator/activity
function_app/function_app.py	Register extract_episodes blueprint
Docs/public_api.md	Rename APIs; document include_episodes + extract_episodes
Docs/local_testing.md	Rename add_cosmos references
Docs/design_patterns.md	Rename APIs in examples
Docs/concepts.md	Update defaults + API rename references
Docs/azure_testing.md	Rename APIs in guidance
CHANGELOG.md	Add new unreleased section; formatting updates
azure/cosmos/agent_memory/thresholds.py	Update defaults; remove dedup vector knobs
azure/cosmos/agent_memory/store/memory_store.py	Rename add_cosmos->upsert_memory; add retrieve_procedures
azure/cosmos/agent_memory/services/init.py	Update store protocol to upsert_memory
azure/cosmos/agent_memory/prompts/extract_procedure.prompty	New prompt for atomic procedures
azure/cosmos/agent_memory/prompts/_schemas.py	Add extract_procedure JSON schema
azure/cosmos/agent_memory/processors/durable.py	Remove episodic no-op warning; durable owns extraction
azure/cosmos/agent_memory/models.py	Add procedure enums/fields; procedural record reshape
azure/cosmos/agent_memory/cosmos_memory_client.py	Rename APIs; add delete helpers; add procedure retrieval/context
azure/cosmos/agent_memory/aio/store/memory_store.py	Async upsert rename; async retrieve_procedures
azure/cosmos/agent_memory/aio/processors/durable.py	Async durable episodic behavior updated
azure/cosmos/agent_memory/aio/cosmos_memory_client.py	Async API rename + delete helpers + procedure retrieval/context
azure/cosmos/agent_memory/_utils.py	Remove vector dedup helpers; simplify container policies
.env.template	Update defaults; add EPISODE_* knobs

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.