Skip to content

fix(generative): send AWS stop_sequences through to the gRPC request - #2131

Open
Anai-Guo wants to merge 2 commits into
weaviate:mainfrom
Anai-Guo:fix-aws-generative-stop-sequences
Open

fix(generative): send AWS stop_sequences through to the gRPC request#2131
Anai-Guo wants to merge 2 commits into
weaviate:mainfrom
Anai-Guo:fix-aws-generative-stop-sequences

Conversation

@Anai-Guo

Copy link
Copy Markdown
Contributor

What

GenerativeConfig.aws_bedrock(...) and aws_sagemaker(...) (the current, non-deprecated AWS generative factories) already accept a stop_sequences argument and store it on the runtime config — but _GenerativeAWS._to_grpc never forwarded it to the gRPC request, so the value was silently dropped and never reached the server.

cfg = GenerativeConfig.aws_bedrock(model="anthropic.claude-3", stop_sequences=["STOP", "\n\n"])
grpc = cfg._to_grpc(_GenerativeConfigRuntimeOptions(return_metadata=True))
# before: grpc.aws.stop_sequences.values == []          <- dropped
# after:  grpc.aws.stop_sequences.values == ['STOP', '\n\n']

Why it's safe to wire unconditionally

The server-side GenerativeAWS message carries stop_sequences in every vendored proto version shipped in this repo (v4216, v5261, v6300), so it can be sent the same way max_tokens already is — no version guard needed. The # TODO - add top_k, top_p & stop_sequences ... comment was stale for stop_sequences; I've narrowed it to the two fields the proto still lacks (top_k, top_p).

Changes

  • _GenerativeAWS._to_grpc: forward stop_sequences=_to_text_array(self.stop_sequences).
  • Narrow the stale TODO to top_k & top_p.
  • Expose stop_sequences on the deprecated aws() factory for parity (it hard-coded stop_sequences=None).
  • Extend the existing AWS _to_grpc test to assert stop_sequences round-trips.

test/collection/test_classes_generative.py passes (20 passed); ruff format --check and ruff check are clean.

🤖 Generated with Claude Code

`GenerativeConfig.aws_bedrock`/`aws_sagemaker` already accept a
`stop_sequences` argument and store it on the runtime config, but
`_GenerativeAWS._to_grpc` never forwarded it, so the value was silently
dropped and never reached the server. The server-side `GenerativeAWS`
proto has carried `stop_sequences` across every vendored version
(v4216/v5261/v6300), so the field can be wired unconditionally the same
way `max_tokens` already is.

Also expose `stop_sequences` on the (deprecated) `aws()` factory for
parity, and narrow the stale TODO to the two fields the proto still
lacks (`top_k`, `top_p`).

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

target_model: Optional[str] = None,
target_variant: Optional[str] = None,
temperature: Optional[float] = None,
stop_sequences: Optional[List[str]] = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this field here? We do not want to update deprecated factories

@shashvat-singham shashvat-singham left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this against the vendored proto and the sibling providers rather than just the diff — it holds up on both counts.

The field really is there, so removing it from the TODO is correct:

>>> [f.name for f in generative_pb2.GenerativeAWS.DESCRIPTOR.fields]
['model', 'temperature', 'service', 'region', 'endpoint', 'target_model',
 'target_variant', 'images', 'image_properties', 'max_tokens', 'stop_sequences']

and the trimmed comment is still accurate for what's left — top_k and top_p are genuinely absent from GenerativeAWS, so "add top_k & top_p here when added to server-side proto" is the right residual.

The part that convinced me this is a real bug rather than an intentional omission: AWS was the only provider passing a hardcoded stop_sequences=None. Seven other generative factories already take stop_sequences: Optional[List[str]] = None and forward it, and _GenerativeAWS already declared stop_sequences: Optional[List[str]] on the dataclass — so the plumbing existed on both ends and only the factory argument and the _to_grpc line were missing. That's a straightforward oversight, and this closes it consistently with how the others are written.

test_generative_parameters_images_parsing exercises the _to_grpc path with stop_sequences=["\n"] and asserts the expected base_pb2.TextArray(values=["\n"]), which is the right level to test it at.

Looks good to me.

Per review: revert the stop_sequences addition to the deprecated aws()
factory and move test coverage to aws_sagemaker. The core fix (sending
stop_sequences through _to_grpc for GenerativeAWS) is unchanged, so
aws_bedrock/aws_sagemaker still carry the field to the proto.
@Anai-Guo

Copy link
Copy Markdown
Contributor Author

Done — reverted the stop_sequences addition on the deprecated aws() factory (it now passes stop_sequences=None again, as before). The core fix is unchanged: _GenerativeAWS._to_grpc now forwards stop_sequences to the proto, so the non-deprecated aws_bedrock / aws_sagemaker factories (which already accept the field) carry it through. Moved the test case over to aws_sagemaker so the path stays covered without touching the deprecated factory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants