Getting started on the Duke Computer Cluster.

Watch the video below for an explanation of the Duke Computing Cluster

Note that the output you get from running this code should be the same as in the demo in the Introduction to FEM tab where you do the same process in the LS Prepost GUI (graphical user interface)

Please watch the video for full steps, but abbreviated steps are below:

1. Set up your DCC acccount.

   1. If you haven't already, go to: https://oit.duke.edu/what-we-do/applications/account-self-service

   2. Go to Account self service

   3. under Advanced user options > change your UNIX shell

   4. change your current shell to /bin/bash

2. Log into the DCC

   1. In your command line interface (CLI) type: ssh yournetid@dcc-login.oit.duke.edu

   2. Log in with your Duke NetID and password

3. Make and navigate to your work directory

   1. in the CLI type: cd /work

   2. in the CLI type: mkdir yourNETID

   3. cd into your new folder and ls to make sure it's empty

Editing the .bashrc file

We need to edit some set up files so that we can access the LS-DYNA license server

• cd $HOME

• ls -a

  • this should take you to the home location you started with. the ls -a command allows us to see files that are usually hidden (files that start with a period are hidden). We sill now edit the .bashrc file, a shell script that Bash runs whenever it starts interactively. We can customize certain paths we want always accessed here. Similar to adding paths at the start of a MATLAB script.

• vim .bashrc

• i to switch to interactive mode

• Copy or type the following command under user defined functions in your .bashrc file

  • #LSTC env vars

  • LSTC_LICENSE=network

  • LSTC_LICENSE_SERVER=durmstrang.egr.duke.edu

  • export LSTC_LICENSE LSTC_LICENSE_SERVER

• escape to exit interactive mode then :wq to write and quit vim and return you to the CLI

• You will need to log fully out from the DCC then log back in for your .bashrc changes to take effect

Python Virtual Environment Setup and Installing the Git repository

• cd /work/yournetID

• module load Python/3.8.1

• python3.8 -m venv .venv

  • This creates a directory .venv in your current working directory to set up the virtual environment

  • Go to github.com/mlp6/fem to see current version requirements

• source .venv/bin/activate

  • You should see (.venv) at the begining of your CLI now

• Next we need to install all the python scripts associated

• pip install git+https://github.com/mlp6/fem.git

  • this is the lightest weight version that doesn't bring the example files along for the ride so you would need to bring them yourself (see video in Pt 2 on transferring files on and off the DCC)

• pip install -e git+https://github.com/mlp6/fem.git#egg=fem

  • this brings all the example files along and installs in editable mode

• the fem repo should be installed in /work/yournetID/.venv/src/fem

• To exit your venv at any time you can just type deactivate

Instructions

Following along with the video below, run your first finite element model on the DCC, specifically the compress cube example from the FEM repository. It is highly recommended that you have success with this process before you move on to more complex finite element models.

Commands Specific to Running Compress Cube

• Make a folder to put things in

  • cd /work/yournetID

  • mkdir FEMTesting

  • cd FEMTesting

  • mkdir CompressCube

• To copy example files

  • cd /work/yournetID/.venv/src/fem/examples/compress_cube

  • cp CompressElasticCubeExplicit.dyn /work/yournetID/FEMTesting/CompressCube

  • cp run.py /work/yournetID/FEMTesting/CompressCube

  • cd /work/yournetID/FEMTEsting/CompressCube

• To edit the run.py to point at the singularity container of the LS-DYNA solver

  • vim run.py

  • i to get to interactive mode

  • replace the line that starts with system('ls-dyna-d'....with the following

  • system('singularity exec -p -B /work/yournetId/FEMTesting/CompressCube /opt/apps/staging/ls-dyna-singularity/ls-dyna.sif ls-dyna-d ncpu={} i={} memory = 600000000'.fomat(NTASKS, DYNADECK))

  • escape then :wq to exit interactive mode sand save changes

• To create a bash script to call this python script

  • touch CCBashLauncher.sh

  • vim CCBashLauncher.sh

  • Add the following lines via insert mode

    • #!/bin/bash

    • #SBATCH -o slurm.%A.out #STDOUT

    • #SBATCH -e slurm.%A.out #STDERR

    • #SBATCH --mem=16G

    • #SBATCH --partition=ultrasound

    • #SBATCH --exclude==dcc-ultrasound-01

    • #SBATCH --cpus-per-task=8

    • date

    • hostname

    • module load Python/3.8.1

    • source /work/aek27/.venv/bin/activate

    • python run.py

• To launch a script

  • sbatch CCBashLauncher.sh

• To check on your jobs

  • squeue -u yournetID to see just your jobs

  • squeue -p ultrasound to see jobs by all users of the ultrasound partition only

  • cat slurm.JOBIDNUMBER.out