S3 file storage
To share files S3 (Simple Storage Service) is used. In this tutorial we use MinIO client to work with it.
You can get access, browse and download files using your browser, using this link:
The LOGIN and PASSWORD for read access are available for all members (but can’t be published at open places like this site). Please ask on #software-helpdesk Slack channel or email one of the SWG members.
One can access S3 directly using TFile like this
TFile* file = TFile::Open("s3https://dtn01.sdcc.bnl.gov:9000/eictest/ATHENA/<path>.root","AUTH=<login>:<password>");
Or set environment variables S3_ACCESS_KEY and S3_SECRET_KEY instead of using AUTH and putting login and password to code;
export S3_ACCESS_KEY=<login>
export S3_SECRET_KEY=<password>
If you encounter Forbidden (403) errors at BNL RCF, try adding the NOPROXY option:
TFile* file = TFile::Open("s3https://dtn01.sdcc.bnl.gov:9000/eictest/ATHENA/<path>.root","NOPROXY AUTH=<login>:<password>");
XRootD access
One can utilize XRootD server to get access to files without credentials using:
root://sci-xrootd.jlab.org//osgpool/eic/ATHENA/<path to file>
std::string fileName = "RECO/JETS/crossDivNrgCrab/DIS_NC_Q2gt10_crossDivNrgCrab_25mRad_18x275_v1.0001.root";
TFile* file = TFile::Open("root://sci-xrootd.jlab.org//osgpool/eic/ATHENA/" + fileName);
// ...
(!) Currently (July 2021) only RECO is on xrootd (since we only have a 2.5 TB limit there)
Installation
Installing MinIO client:
wget https://dl.min.io/client/mc/release/linux-amd64/mc
mkdir -p $HOME/bin && mv mc $HOME/bin/mc && chmod +x $HOME/bin/mc
alias mc="$HOME/bin/mc"
mc --autocompletion
> You may name the command diffirently if mc aka midnight commander which also uses mc is installed
Getting access
Public read-only access
mc config host add S3 https://dtn01.sdcc.bnl.gov:9000 <login> <password>
Private write-enabled access
mc config host add S3rw https://dtn01.sdcc.bnl.gov:9000/ $user $key
Private write-enabled access at BNL
mc config host add S3rw http://eicoss01.sdcc.bnl.local:9000 $user $key
Operations
Listing contents (the eictest is historical and may change)
mc tree S3/eictest/ATHENA
mc ls S3/eictest/ATHENA
mc find S3/eictest/ATHENA --name '*.hepmc*'
mc du S3/eictest/ATHENA
Copying contents out
mc cp --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc .
Mirroring directories
mc mirror --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/ crossDivNrgCrab/
Streaming contents out
mc cat --insecure S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | head -n 20
mc cat S3/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz | gunzip -c | head -n 20
Copying contents in
mc cp /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_* S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab
Streaming contents in
cat /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc
gzip -c /gpfs02/eic/bpage/home/eicBeamSimu/Pythia8/headonTestJin/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc | mc pipe S3rw/eictest/ATHENA/EVGEN/JETS/crossDivNrgCrab/test_crossDivNrgCrab_25mRad_18x275_v1.hepmc.gz