s3inbox Service
The s3inbox
proxies uploads to the an S3 compatible storage backend. Users are authenticated with a JWT instead of access_key
and secret_key
used normally for S3
.
Service Description
The s3inbox
proxies uploads to an S3 compatible storage backend.
- Parses and validates the JWT token (
access_token
in the S3 config file) against the public keys, either locally provisioned or from OIDC JWK endpoints. - If the token is valid the file is passed on to the S3 backend
- The file is registered in the database
- The
inbox-upload
message is sent to theinbox
queue, with thesub
field from the token as theuser
in the message. If this fails an error will be written to the logs.
Communication
s3inbox
proxies uploads to inbox storage.s3inbox
inserts file information in the database using theRegisterFile
database function and marks it as uploaded in thefile_event_log
s3inbox
writes messages to one RabbitMQ queue (commonly:inbox
).
Configuration
There are a number of options that can be set for the s3inbox
service.
These settings can be set by mounting a yaml-file at /config.yaml
with settings.
ex.
log:
level: "debug"
format: "json"
They may also be set using environment variables like:
export LOG_LEVEL="debug"
export LOG_FORMAT="json"
Server settings
These settings control the TLS status and where the service gets the public keys to validate the JWT tokens.
SERVER_CERT
: path to the x509 certificate used by the serviceSERVER_KEY
: path to the x509 private key used by the serviceSERVER_JWTPUBKEYPATH
: full path to the folder containing public keys used to validate JWT tokensSERVER_JWTPUBKEYURL
: URL to OIDC JWK endpoint
RabbitMQ broker settings
These settings control how verify connects to the RabbitMQ message broker.
BROKER_HOST
: hostname of the RabbitMQ serverBROKER_PORT
: RabbitMQ broker port (commonly:5671
with TLS and5672
without)BROKER_QUEUE
: message queue to read messages from (commonly:archived
)BROKER_ROUTINGKEY
: Routing key for publishing messages (commonly:verified
)BROKER_USER
: username to connect to RabbitMQBROKER_PASSWORD
: password to connect to RabbitMQBROKER_PREFETCHCOUNT
: Number of messages to pull from the message server at the time (default to2
)
PostgreSQL Database settings
DB_HOST
: hostname for the postgresql databaseDB_PORT
: database port (commonly:5432
)DB_PASSWORD
: password for the databaseDB_DATABASE
: database nameDB_SSLMODE
: The TLS encryption policy to use for database connections, valid options are:disable
allow
prefer
require
verify-ca
verify-full
More information is available in the postgresql documentation
Note that if DB_SSLMODE
is set to anything but disable
, then DB_CACERT
needs to be set, and if set to verify-full
, then DB_CLIENTCERT
, and DB_CLIENTKEY
must also be set.
DB_CLIENTKEY
: key-file for the database client certificateDB_CLIENTCERT
: database client certificate fileDB_CACERT
: Certificate Authority (CA) certificate for the database to use
Storage settings
INBOX_TYPE
: Valid value isS3
INBOX_URL
: URL to the S3 systemINBOX_ACCESSKEY
: The S3 access and secret key are used to authenticate to S3, more info at AWSINBOX_SECRETKEY
: The S3 access and secret key are used to authenticate to S3, more info at AWSINBOX_BUCKET
: The S3 bucket to use as the storage rootINBOX_PORT
: S3 connection port (default:443
)INBOX_REGION
: S3 region (default:us-east-1
)INBOX_CHUNKSIZE
: S3 chunk size for multipart uploads.INBOX_CACERT
: Certificate Authority (CA) certificate for the storage system, this is only needed if the S3 server has a certificate signed by a private entity
Logging settings
LOG_FORMAT
can be set to “json” to get logs in json format. All other values result in text loggingLOG_LEVEL
can be set to one of the following, in increasing order of severity:trace
debug
info
warn
(orwarning
)error
fatal
panic