Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
O
OnedataSim
Manage
Activity
Members
Labels
Plan
Issues
0
Issue boards
Milestones
Wiki
Code
Merge requests
0
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
LAGO
Software
OnedataSim
Commits
f6ff3f52
Unverified
Commit
f6ff3f52
authored
2 years ago
by
AJRubio-Montero
Committed by
GitHub
2 years ago
Browse files
Options
Downloads
Patches
Plain Diff
Update README.md
parent
cbdb7227
No related branches found
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
README.md
+36
-20
36 additions, 20 deletions
README.md
with
36 additions
and
20 deletions
README.md
+
36
−
20
View file @
f6ff3f52
...
@@ -7,15 +7,15 @@
...
@@ -7,15 +7,15 @@
-
onedatasim-s1 image:
-
onedatasim-s1 image:
[

](https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/onedataSim/job/build-S1/)
[

](https://jenkins.eosc-synergy.eu/job/eosc-synergy-org/job/onedataSim/job/build-S1/)
## About
## About
onedataSim standardises the simulations and their analysis in LAGO Collaboration
onedataSim standardises the simulations and their analysis in LAGO Collaboration
to curate, re-use and publish the results, following the Data Management Plan
to curate, re-use and publish the results, following the Data Management Plan
(DMP) established
[
(https://lagoproject.github.io/DMP/)
](
https://lagoproject.github.io/DMP/
)
. For this purpose,
(DMP) established
onedataSim packets ARTI and related software into a Docker image, giving
[
(https://lagoproject.github.io/DMP/)
](
https://lagoproject.github.io/DMP/
)
.
researchers the advantage of obtaining results on any plataform and publishing
For this purpose, onedataSim packets ARTI and related software into a Docker
them on LAGO repositories.
image, giving researchers the advantage of obtaining results on any plataform
and publishing them on LAGO repositories.
### Citing
### Citing
...
@@ -193,6 +193,7 @@ export ONEPROVIDER="ceta-ciemat-01.datahub.egi.eu"
...
@@ -193,6 +193,7 @@ export ONEPROVIDER="ceta-ciemat-01.datahub.egi.eu"
```
```
2.
Showing parameters:
2.
Showing parameters:
```
sh
```
sh
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
...
@@ -204,7 +205,9 @@ sudo docker run --privileged -e ONECLIENT_ACCESS_TOKEN=$TOKEN \
...
@@ -204,7 +205,9 @@ sudo docker run --privileged -e ONECLIENT_ACCESS_TOKEN=$TOKEN \
```
sh
```
sh
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-it
lagocollaboration/onedatasim-s0:dev bash
-lc
"do_sims_onedata.py -t 10 -u 0000-0001-6497-753X -s and -k 2.0e2 -h QGSII -x"
-it
lagocollaboration/onedatasim-s0:dev
\
bash
-lc
"do_sims_onedata.py -t 10
\
-u 0000-0001-6497-753X -s and -k 2.0e2 -h QGSII -x"
```
```
3.
Executing on a multi-processor server.
3.
Executing on a multi-processor server.
...
@@ -216,7 +219,8 @@ instantiated with enough procesors memory and disk, you only need add the
...
@@ -216,7 +219,8 @@ instantiated with enough procesors memory and disk, you only need add the
```
sh
```
sh
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-it
<container name> bash
-lc
"do_sims_onedata.py <other ARTI do_* params> -j <procs>"
-it
<container name> bash
-lc
"do_sims_onedata.py
\
<other ARTI do_* params> -j <procs>"
```
```
### Analysing S0 datasets (generating S1 data)
### Analysing S0 datasets (generating S1 data)
...
@@ -233,7 +237,8 @@ export ONEPROVIDER="ceta-ciemat-01.datahub.egi.eu"
...
@@ -233,7 +237,8 @@ export ONEPROVIDER="ceta-ciemat-01.datahub.egi.eu"
```
sh
```
sh
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-it
lagocollaboration/onedatasim-s1:dev bash
-lc
"do_showers_onedata.py -?"
-it
lagocollaboration/onedatasim-s1:dev
\
bash
-lc
"do_showers_onedata.py -?"
```
```
3.
Executing an analysis:
3.
Executing an analysis:
...
@@ -241,7 +246,8 @@ sudo docker run --privileged -e ONECLIENT_ACCESS_TOKEN=$TOKEN \
...
@@ -241,7 +246,8 @@ sudo docker run --privileged -e ONECLIENT_ACCESS_TOKEN=$TOKEN \
```
sh
```
sh
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-it
<container name> bash
-lc
"do_showers_onedata.py -o XXXX -u 0000-0001-6497-753X"
-it
<container name> bash
-lc
"do_showers_onedata.py
\
-o XXXX -u 0000-0001-6497-753X"
```
```
## Advanced use cases
## Advanced use cases
...
@@ -289,6 +295,7 @@ and other schedulers (Kubernetes).
...
@@ -289,6 +295,7 @@ and other schedulers (Kubernetes).
OpenNebula, XenServer, VMware, etc).
OpenNebula, XenServer, VMware, etc).
2.
Example for an Slurm instantiated on EOSC resources (pre-configured by IM):
2.
Example for an Slurm instantiated on EOSC resources (pre-configured by IM):
You can access to head node through SSH, using
``cloudadm``
account, but then
You can access to head node through SSH, using
``cloudadm``
account, but then
you can gain root privileges with
``sudo``
. Slurm and a directory shared by NFS
you can gain root privileges with
``sudo``
. Slurm and a directory shared by NFS
are already configured (/home), but some configruation has to be done: to share
are already configured (/home), but some configruation has to be done: to share
...
@@ -312,7 +319,7 @@ sbatch simulation.sbatch
...
@@ -312,7 +319,7 @@ sbatch simulation.sbatch
```
```
A simulation.sbatch file for testing functionality can be one that will write
A simulation.sbatch file for testing functionality can be one that will write
the allowed parameters in
<job
number
>
.log:
the allowed parameters in
``
<job number>.log
``
:
```
sh
```
sh
#!/bin/bash
#!/bin/bash
...
@@ -326,7 +333,9 @@ sudo apt-get install -y docker.io
...
@@ -326,7 +333,9 @@ sudo apt-get install -y docker.io
sudo
docker stop
$(
docker ps
-aq
)
sudo
docker stop
$(
docker ps
-aq
)
sudo
docker
rm
$(
docker ps
-aq
)
sudo
docker
rm
$(
docker ps
-aq
)
sudo
docker load
-i
-o
/home/cloudadm/onedatasim-s0.tar
sudo
docker load
-i
-o
/home/cloudadm/onedatasim-s0.tar
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
-i
onedatasim-s0:dev bash
-lc
"do_sims_onedata.py -?"
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
$TOKEN
\
-e
ONECLIENT_PROVIDER_HOST
=
$ONEPROVIDER
\
-i
onedatasim-s0:dev bash
-lc
"do_sims_onedata.py -?"
```
```
## Instructions only for developers
## Instructions only for developers
...
@@ -378,22 +387,27 @@ sudo docker build --build-arg ONEDATASIM_BRANCH="dev-ajrubio-montero" \
...
@@ -378,22 +387,27 @@ sudo docker build --build-arg ONEDATASIM_BRANCH="dev-ajrubio-montero" \
### Logging into container for developing purposes
### Logging into container for developing purposes
1.
Runing scripts & attaching a local directory at login.
1.
Runing scripts & attaching a local directory at login.
To log into the container only has to run bash without parameters,
To log into the container only has to run bash without parameters,
positioned alwasy at the end of the command. Additionally, You can mount
positioned alwasy at the end of the command. Additionally, You can mount
a local directory inside the container the with the parameter
a local directory inside the container the with the
**--volume \<local path\>:\<container path\>**
. For example:
parameter
**--volume \<local path\>:\<container path\>**
.
For example:
```
sh
```
sh
[
pepe@mypc tmp]#
ls
/home/pepe/workspace
[
pepe@mypc tmp]#
ls
/home/pepe/workspace
onedataSim samples geant4-dev
onedataSim samples geant4-dev
[
pepe@mypc tmp]#
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
"MDAxY2xv...iXm8jowGgo"
\
[
pepe@mypc tmp]#
sudo
docker run
--privileged
\
-e
ONECLIENT_ACCESS_TOKEN
=
"MDAxY2xv...iXm8jowGgo"
\
-e
ONECLIENT_PROVIDER_HOST
=
"mon01-tic.ciemat.es"
\
-e
ONECLIENT_PROVIDER_HOST
=
"mon01-tic.ciemat.es"
\
--volume
/home/pepe/workspace:/root
-it
lagocontainer:0.0.1 bash
--volume
/home/pepe/workspace:/root
\
-it
lagocontainer:0.0.1 bash
[
root@c42dc622f7eb run]#
ls
/root
[
root@c42dc622f7eb run]#
ls
/root
onedataSim samples geant4-dev
onedataSim samples geant4-dev
```
```
2.
Explore OneData repository within the container.
2.
Explore OneData repository within the container.
Firstly test if the repository is already mounted and force mount if necessary:
Firstly test if the repository is already mounted and force mount if necessary:
```
sh
```
sh
...
@@ -444,17 +458,19 @@ development. For this purpose you should the suitable OneData provider and use
...
@@ -444,17 +458,19 @@ development. For this purpose you should the suitable OneData provider and use
the the
``--onedata_path``
parameter to select the correct path.
the the
``--onedata_path``
parameter to select the correct path.
For
``test8``
, you should choose ceta-ciemat-
**02**
.datahub.egi.eu and any
For
``test8``
, you should choose ceta-ciemat-
**02**
.datahub.egi.eu and any
directory
<dir>
under
the
``--onedata_path /mnt/datahub.egi.eu/test8/<dir>``
directory
``
<dir>
``
under
path:
the
``--onedata_path /mnt/datahub.egi.eu/test8/<dir>``
path:
```
sh
```
sh
export
TOKEN
=
"MDAxY...LAo"
export
TOKEN
=
"MDAxY...LAo"
export
ONEPROVIDER
=
"ceta-ciemat-02.datahub.egi.eu"
export
ONEPROVIDER
=
"ceta-ciemat-02.datahub.egi.eu"
[
pepe@mypc tmp]#
sudo
docker run
--privileged
-e
ONECLIENT_ACCESS_TOKEN
=
"
$TOKEN
"
\
[
pepe@mypc tmp]#
sudo
docker run
--privileged
\
-e
ONECLIENT_ACCESS_TOKEN
=
"
$TOKEN
"
\
-e
ONECLIENT_PROVIDER_HOST
=
"
$ONEPROVIDER
"
\
-e
ONECLIENT_PROVIDER_HOST
=
"
$ONEPROVIDER
"
\
-it
lagocollaboration/onedatasim-s0:dev bash
-it
lagocollaboration/onedatasim-s0:dev bash
[
root@9db2578a3e28 run]# do_sims_onedata.py
-t
13
-u
0000-0001-6497-753X
-s
and
-k
2.0e2
-h
QGSII
-x
--onedata_path
/mnt/datahub.egi.eu/test8/LAGOSIM_test_20220210
-j
4
[
root@9db2578a3e28 run]# do_sims_onedata.py
-t
13
-u
0000-0001-6497-753X
\
-s
and
-k
2.0e2
-h
QGSII
-x
--onedata_path
\
/mnt/datahub.egi.eu/test8/LAGOSIM_test_20220210
-j
4
```
```
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment