-
Notifications
You must be signed in to change notification settings - Fork 561
feat(cohorts): Amplitude cohort sync endpoints and sync keys #8290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gagantrivedi
wants to merge
13
commits into
main
Choose a base branch
from
feat/cohort-sync-amplitude
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,278
−14
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
e75ac42
feat(cohorts): add Amplitude cohort sync endpoints and sync keys
gagantrivedi 5bf748a
chore: Update documentation artefacts
flagsmith-engineering[bot] ec31f2b
fix(cohorts): audit source-created cohorts and drop the machine user
gagantrivedi 129440f
chore: Update documentation artefacts
flagsmith-engineering[bot] 3cbb0ce
refactor(cohorts): inline membership upsert batch size
gagantrivedi febe7eb
refactor(cohorts): drop redundant sync key docstring
gagantrivedi f142565
refactor(cohorts): drop sync key edge check and module docstring
gagantrivedi 9254c6f
feat(cohorts): document the cohort sync key auth scheme
gagantrivedi 3964c41
fix(cohorts): drop the edge gate from the Amplitude endpoint
gagantrivedi 040cac7
chore: Update documentation artefacts
flagsmith-engineering[bot] c7dfa1d
fix(cohorts): batch membership removals and split delta events
gagantrivedi 1370c09
fix(cohorts): reject identifiers over the 1024-byte edge limit
gagantrivedi ee9cfda
chore: Update documentation artefacts
flagsmith-engineering[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| import typing | ||
| from contextlib import suppress | ||
|
|
||
| from django.contrib.auth.models import AnonymousUser | ||
| from rest_framework import authentication, exceptions | ||
| from rest_framework.request import Request | ||
|
|
||
| from cohorts.models import CohortSyncKey | ||
|
|
||
|
|
||
| class CohortSyncKeyAuthentication(authentication.BaseAuthentication): | ||
| def authenticate( | ||
| self, request: Request | ||
| ) -> tuple[AnonymousUser, CohortSyncKey] | None: | ||
| header = request.headers.get("Authorization", "") | ||
| if not header.startswith("Bearer "): | ||
| return None | ||
|
|
||
| with suppress(CohortSyncKey.DoesNotExist): | ||
| key = typing.cast( | ||
| CohortSyncKey, | ||
| CohortSyncKey.objects.get_from_key(header.removeprefix("Bearer ")), | ||
| ) | ||
| if not key.has_expired: | ||
| # No person is acting here, so no user is returned: the key | ||
| # alone carries authority, and audit trails record the source | ||
| # rather than a user. | ||
| return AnonymousUser(), key | ||
|
|
||
| raise exceptions.AuthenticationFailed("Valid cohort sync key not found.") | ||
|
|
||
| def authenticate_header(self, request: Request) -> str: | ||
| # Makes missing or invalid credentials a 401 rather than DRF's default 403. | ||
| return "Bearer" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Generated by Django 5.2.16 on 2026-08-14 08:31 | ||
|
|
||
| import django.db.models.deletion | ||
| from django.conf import settings | ||
| from django.db import migrations, models | ||
|
|
||
|
|
||
| class Migration(migrations.Migration): | ||
|
|
||
| dependencies = [ | ||
| ("cohorts", "0002_cohort_deletion_requested_at"), | ||
| ("environments", "0039_use_no_ssrf_url_field"), | ||
| migrations.swappable_dependency(settings.AUTH_USER_MODEL), | ||
| ] | ||
|
|
||
| operations = [ | ||
| migrations.AlterField( | ||
| model_name="cohort", | ||
| name="source_type", | ||
| field=models.CharField( | ||
| choices=[("csv", "CSV"), ("amplitude", "Amplitude")], | ||
| default="csv", | ||
| max_length=50, | ||
| ), | ||
| ), | ||
| migrations.CreateModel( | ||
| name="CohortSyncKey", | ||
| fields=[ | ||
| ( | ||
| "id", | ||
| models.CharField( | ||
| editable=False, | ||
| max_length=150, | ||
| primary_key=True, | ||
| serialize=False, | ||
| unique=True, | ||
| ), | ||
| ), | ||
| ("prefix", models.CharField(editable=False, max_length=8, unique=True)), | ||
| ("hashed_key", models.CharField(editable=False, max_length=150)), | ||
| ("created", models.DateTimeField(auto_now_add=True, db_index=True)), | ||
| ( | ||
| "name", | ||
| models.CharField( | ||
| default=None, | ||
| help_text="A free-form name for the API key. Need not be unique. 50 characters max.", | ||
| max_length=50, | ||
| ), | ||
| ), | ||
| ( | ||
| "revoked", | ||
| models.BooleanField( | ||
| blank=True, | ||
| default=False, | ||
| help_text="If the API key is revoked, clients cannot use it anymore. (This cannot be undone.)", | ||
| ), | ||
| ), | ||
| ( | ||
| "expiry_date", | ||
| models.DateTimeField( | ||
| blank=True, | ||
| help_text="Once API key expires, clients cannot use it anymore.", | ||
| null=True, | ||
| verbose_name="Expires", | ||
| ), | ||
| ), | ||
| ( | ||
| "created_by", | ||
| models.ForeignKey( | ||
| blank=True, | ||
| null=True, | ||
| on_delete=django.db.models.deletion.SET_NULL, | ||
| to=settings.AUTH_USER_MODEL, | ||
| ), | ||
| ), | ||
| ( | ||
| "environment", | ||
| models.ForeignKey( | ||
| on_delete=django.db.models.deletion.CASCADE, | ||
| related_name="cohort_sync_keys", | ||
| to="environments.environment", | ||
| ), | ||
| ), | ||
| ], | ||
| options={ | ||
| "verbose_name": "cohort sync key", | ||
| "verbose_name_plural": "cohort sync keys", | ||
| "ordering": ("-created",), | ||
| "abstract": False, | ||
| }, | ||
| ), | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,9 +6,17 @@ | |
| from django.utils import timezone | ||
| from flag_engine.segments.constants import IS_SET | ||
|
|
||
| from audit.constants import SEGMENT_CREATED_MESSAGE | ||
| from audit.models import AuditLog | ||
| from audit.related_object_type import RelatedObjectType | ||
| from cohorts.constants import COHORT_MEMBERSHIP_APPLY_BATCH_SIZE | ||
| from cohorts.metrics import flagsmith_cohorts_membership_deltas_applied_total | ||
| from cohorts.models import Cohort, CohortMembership, CohortMembershipState | ||
| from cohorts.models import ( | ||
| Cohort, | ||
| CohortMembership, | ||
| CohortMembershipState, | ||
| CohortSourceType, | ||
| ) | ||
| from core.dataclasses import AuthorData | ||
| from environments.identities.system_traits import ( | ||
| set_system_trait, | ||
|
|
@@ -81,6 +89,7 @@ def create_cohort( | |
| environment: "Environment", | ||
| name: str, | ||
| description: str | None = None, | ||
| source_type: CohortSourceType = CohortSourceType.CSV, | ||
| ) -> Cohort: | ||
| with transaction.atomic(): | ||
| segment = Segment.objects.create( | ||
|
|
@@ -90,7 +99,9 @@ def create_cohort( | |
| managed_by=SegmentManagedBy.COHORT, | ||
| ) | ||
| rule = SegmentRule.objects.create(segment=segment, type=SegmentRule.ALL_RULE) | ||
| cohort: Cohort = Cohort.objects.create(environment=environment, segment=segment) | ||
| cohort: Cohort = Cohort.objects.create( | ||
| environment=environment, segment=segment, source_type=source_type | ||
| ) | ||
| Condition.objects.create( | ||
| rule=rule, | ||
| operator=IS_SET, | ||
|
|
@@ -108,6 +119,86 @@ def create_cohort( | |
| return cohort | ||
|
|
||
|
|
||
| def create_cohort_for_source( | ||
| *, | ||
| environment: "Environment", | ||
| name: str, | ||
| source_type: CohortSourceType, | ||
| ) -> Cohort: | ||
| """Create a cohort on behalf of an external source, where no Flagsmith | ||
| user is acting.""" | ||
| cohort = create_cohort(environment=environment, name=name, source_type=source_type) | ||
| # Nothing records a user for these calls, so the audit log that Flagsmith | ||
| # derives from historical records is skipped — and with it the environment | ||
| # document rebuild that makes the new segment visible to SDKs. Write the | ||
| # record here instead, naming the source that asked for the cohort. | ||
|
Comment on lines
+133
to
+134
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So if the log creation fails, the created cohort won't be part of the document until next rebuild ? Should we group them in the same transaction? |
||
| AuditLog.objects.create( | ||
| environment=environment, | ||
| project=environment.project, | ||
| related_object_id=cohort.segment_id, | ||
| related_object_type=RelatedObjectType.SEGMENT.name, | ||
| log=( | ||
| f"{SEGMENT_CREATED_MESSAGE % cohort.segment.name} " | ||
| f"(via {CohortSourceType(source_type).label} cohort sync)" | ||
| ), | ||
| ) | ||
| return cohort | ||
|
gagantrivedi marked this conversation as resolved.
|
||
|
|
||
|
|
||
| def add_cohort_members(cohort: Cohort, identifiers: "typing.Iterable[str]") -> None: | ||
| from cohorts.tasks import apply_cohort_membership_deltas | ||
|
|
||
| rows = [ | ||
| CohortMembership(cohort=cohort, identifier=identifier) | ||
| for identifier in set(identifiers) | ||
| ] | ||
| with transaction.atomic(): | ||
| # Re-adding a member is a no-op end to end: an applied row flips back | ||
| # to pending and the identity write it triggers is idempotent. | ||
| CohortMembership.objects.bulk_create( | ||
| rows, | ||
| # Postgres rejects a statement carrying more than 65535 bind | ||
| # parameters, which a single large batch would exceed. | ||
| batch_size=1000, | ||
| update_conflicts=True, | ||
| unique_fields=["cohort", "identifier"], | ||
| update_fields=["state", "updated_at"], | ||
| ) | ||
| apply_cohort_membership_deltas.delay(kwargs={"cohort_id": cohort.id}) | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| logger.info( | ||
| "membership.adds_received", | ||
| cohort__id=cohort.id, | ||
| environment__id=cohort.environment_id, | ||
| deltas__count=len(rows), | ||
| ) | ||
|
|
||
|
|
||
| def remove_cohort_members(cohort: Cohort, identifiers: "typing.Iterable[str]") -> None: | ||
| from cohorts.tasks import apply_cohort_membership_deltas | ||
|
|
||
| unique_identifiers = list(set(identifiers)) | ||
| matched = 0 | ||
| with transaction.atomic(): | ||
| # Postgres rejects a statement carrying more than 65535 bind | ||
| # parameters, which a single large identifier list would exceed. | ||
| for start in range(0, len(unique_identifiers), 1000): | ||
| # Removing a non-member is a no-op: only existing rows flip. | ||
| matched += CohortMembership.objects.filter( | ||
| cohort=cohort, | ||
| identifier__in=unique_identifiers[start : start + 1000], | ||
| ).update( | ||
| state=CohortMembershipState.PENDING_REMOVE, updated_at=timezone.now() | ||
| ) | ||
| apply_cohort_membership_deltas.delay(kwargs={"cohort_id": cohort.id}) | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| logger.info( | ||
| "membership.removals_received", | ||
| cohort__id=cohort.id, | ||
| environment__id=cohort.environment_id, | ||
| deltas__count=len(unique_identifiers), | ||
| members__matched=matched, | ||
| ) | ||
|
|
||
|
|
||
| def delete_cohort(cohort: Cohort) -> None: | ||
| from cohorts.tasks import apply_cohort_membership_deltas | ||
|
|
||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.