Data Upload Process

To ingest data into CSFEye it is required to copy your files to a dedicated S3 bucket, shared by Synamedia. The below instructions describe the process of obtaining temporary AWS credentials to perform,

1. The data upload

2. The relevant AWS CLI commands to copy data once authenticated

Only csv files are supported. Files need to comply with the CSFEye Data Requirements document.

Make sure you have the required dependencies installed

If you are uploading data from a Linux or MacOS machine, you will need to have the following dependencies installed:

• The AWS CLI (https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)

• curl (https://curl.se/)

• jq (https://stedolan.github.io/jq/)

If you are uploading data from a Windows machine, you will need to have the following dependencies installed:

• The AWS CLI (https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)

Note: The helper script described in step 2 will check for the presence of these dependencies and exit if they are not found. 

Download and prepare the helper script

In the same OneDrive folder as these instructions, you will find three files:

• csfeye_s3.sh - This is a Bash script which can be used if you are trying to upload data from a Linux or MacOS machine

• csfeye_s3.ps1 - This is a PowerShell script which can be used if you are trying to upload data from a Windows machine

• credentials.json- This contains your personal credentials for data upload as well as the name of the S3 bucket to upload data to

Copy the relevant script for your operating system as well as the credentials.json file to your local machine.

Note: If you are running Linux or MacOS, you will need to make the csfeye_s3.sh script executable by typing "chmod +x csfeye_s3.sh" before you can run it. If you are running Windows you need to ensure the execution policy of your machine allows PowerShell scripts to execute - please contact your IT administrator in the first instance if you are unsure about how to do this.

You will only need to do this step once.

Execute the Helper Script

Scripts can be found in this repository.

The credentials you have been provided will be used to request temporary AWS credentials to allow you to copy data to S3. These credentials will be valid for 1 hour from the time the script is run, and the script can be re-run whenever you need to refresh them.

Linux/MacOS:

Open a terminal and navigate to the directory where the csfeye_s3.sh script is stored, then execute it as follows:

./csfeye_s3.sh

Windows:

Open a PowerShell window (note: not a normal command prompt) and navigate to the directory where the csfeye_s3.ps1 script is stored, then execute it as follows:

.\csfeye_s3.ps1

In both cases, you should see output which looks like this:

Checking dependencies... Obtaining access token... Obtaining AWS credentials... Configuring AWS client... Complete. Credentials valid until 2021-11-23T18:11:11.000Z. Please ensure "--profile csfeye" is appended to all AWS CLI S3 commands, e.g. "aws s3 ls s3://csfeye-abcd1234/ --profile csfeye"

Upload files to S3

Run the following command to upload a single file to S3, replacing FILENAME with the name of the file you wish to upload and S3_BUCKET_NAME with the name of the S3 bucket in the credentials.json file:

aws s3 cp FILENAME s3://S3_BUCKET_NAME/source/ --profile csfeye