Loading documents from S3

How to generate signed URLs

To load documents from S3, you can create a pre-signed URL to access files inside your bucket. Use the following instructions (taken from the AWS documentation).

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

2. In the Buckets list, choose the name of the bucket that contains the object that you want as a pre-signed URL.

3. In the Objects list, select the object for which you want to create a pre-signed URL.

4. On the Actions menu, choose Share with a pre-signed URL.

5. Specify how long you want the pre-signed URL to be valid.

6. Choose 'Create pre-signed URL'.

7. When a confirmation appears, the URL is automatically copied to your clipboard. You will see a button to copy the pre-signed URL if you need to copy it again.

This will generate an URL with the following structure:

https://<BUCKET>.s3.<REGION>.amazonaws.com/<FILE>?response-content-disposition=inline&X-Amz-Security-Token=<TOKEN>

Pre-signed URLs have a maximum duration of 12 hours.

How to add the URL to a Stack AI flow

You can load the document with its signed URL to a URL data loader (see Data Loaders) which will load and parse the PDF to a vector database.

Example of URL node with signed S3 URL connected to a vector database

You can try this flow in the WebScrapper Q&A template.

How to call the document via the API

Go to the Deploy section and you can call specify the signed URL in the inputs of the flow. In CuRL a simple flow works as:

curl "https://www.stack-inference.com/run_deployed_flow?flow_id=<FLOW_ID>&org=<ORG>" \
  -X POST \
  -d '{"in-0": "Summarize this document", "url-0": "https://<BUCKET>.s3.<REGION>.amazonaws.com/<FILE>?response-content-disposition=inline&X-Amz-Security-Token=<TOKEN>"}' \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer XXXXXXXXXXXXX"

For more details on deployment see How to deploy.