Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Output aws_s3: Cross-region not possible when using regional endpoints #2343

Open
Anton-Frank opened this issue Jan 23, 2024 · 0 comments
Open
Labels
aws Issues relating to AWS enhancement

Comments

@Anton-Frank
Copy link
Contributor

Anton-Frank commented Jan 23, 2024

Scenario:
Running Benthos on EKS in eu-central-1 region, inside a private subnet, which only has access to VPC endpoints.
AWS credentials are provided via Service Account Role.
Endpoints are always regional endpoints. There is no chance to access global endpoints. (No NAT Gateways).

Benthos Config snippet:

output:
  label: "output"
  aws_s3:
    bucket: "<my-bucket-name-in-region-eu-west-1>" 
    path: ${!count("files")}-${!timestamp_unix_nano()}.txt
    content_type: application/octet-stream
    max_in_flight: 64
    batching:
      count: 2
      byte_size: 100
    region: 'eu-west-1'

Issue:
Benthos needs to call STS in its own region eu-central-1, but the bucket itself lives in eu-west-1. This results in various errors like
level=error msg="Failed to send message to aws_s3: RequestCanceled: request context canceled\ncaused by: context deadline exceeded"
If the bucket exists in the same region, there is no issue and the output works as expected.

Possible solution:
Optionally configure a separate endpoint for STS (would be in sts.eu-central-1.amazonaws.com in this case) and set the region explicitely for the bucket.

EDIT:
I haven`t checked yet, but I assume, this will be the same behaviour for all AWS endpoints, so the solution probably needs to tackle this within the aws credentials context (https://www.benthos.dev/docs/guides/cloud/aws).

@Jeffail Jeffail added enhancement aws Issues relating to AWS labels Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws Issues relating to AWS enhancement
Projects
None yet
Development

No branches or pull requests

2 participants
  NODES
COMMUNITY 2
Project 5
USERS 1