Amazon S3
Overview
Amazon S3 (Simple Storage Service) is a highly scalable, durable, and secure cloud storage service offered by Amazon Web Services (AWS). It is commonly used for various data storage purposes, including storing server logs.
Using S3 to store server logs provides a reliable, scalable, and secure solution for retaining valuable log data. The centralized storage also simplifies log management, enhances data analysis capabilities, and supports compliance and auditing requirements. AWS’s extensive global infrastructure ensures that your log data is available and accessible from anywhere with low latency, making S3 a popular choice for log storage in the cloud.
Follow the steps below to configure log shipping to Amazon S3.
Step 1 - Credential Procurement
Authentication credentials are required to ship logs to S3.
Follow these steps to obtain the credentials, store them as an AWS Secret, and configure external logging.
-
Refer to these instruction to create a new AWS IAM user in your AWS account.
- Select
Programmatic access
when creating the user and take note of the Access and Secret Key.
- Select
-
This user, at a minimum, must have an associated policy with the
s3:PutObject
action.
Sample AWS Policy (substitute S3_BUCKET_NAME):
- From the Control Plane Console UI, click
Secrets
from the left menu. - Click the
New
button. - Enter a
Name
for the secret, and selectAWS
from the Secret Type list. - Enter the
Access Key
andSecret Key
and clickSave
. - This secret can now be used when configuring logging using the UI Console or CLI.
Step 2 - Configure External Logging
External logging can be configured by using either the UI Console or the CLI.
Enable Logging using the UI Console
- From the Control Plane Console UI, click on
Org
in the left menu. - Click
External Logs
in the middle context menu. - Select
S3
and fill out the required fields. - Select the AWS secret to authenticate to S3. Refer to the credential procurement section to obtain and configure the necessary credentials.
- Click
Save
. - After the configuration is complete, log entries will be available at S3 within a few minutes.
The prefix
will be the folder where the logs will be written.
The folder structure will follow the format:
PREFIX/ORG_NAME/YEAR/MONTH/DAY/HOUR/MINUTE/LOG_FILE.jsonl
The .jsonl file will contain ~1-3k of single line log entries in JSON.
Each entry will contain the following keys:
- time
- log
- location,
- version
- provider
- container
- replica
- workload
- gvc
- org
- stream
Enable Logging using the CLI
The external logging configuration can be created / updated using the CLI’s cpln org patch ORG_NAME -f FILE.yaml
command.
Below is an example Org manifest (in YAML). Edit and save the YAML as a file and use it as an input to the CLI’s cpln org patch ORG_NAME -f FILE.yaml
command.
Refer to the credential procurement section to obtain and configure the necessary credentials.
- Substitute:
ORG_NAME
,S3_BUCKET_NAME
,AWS_SECRET
, andAWS_REGION
.