Reproducibility (paranoia) for Predocs

What are containers and software environments and how can I become a more paranoid bioinformatician?

Michael Hall - Iqbal Group

(Python) virtual environments

$ mkdir paranoid && cd paranoid

$ python3 -c "import numpy; print(numpy.__version__)"
1.18.3 # i.e. whatever is installed in the system wide python

$ python3 -m venv .venv  # create new env

$ which python3  # shows the location of our system python

$ source .venv/bin/activate  # makes our new env the "active" python

$ which python3  # shows the location of our environment python

$ python3 -c "import numpy; print(numpy.__version__)"
Traceback (most recent call last):
  File "np.py", line 2, in <module>
    import numpy
ModuleNotFoundError: No module named 'numpy'

$ pip install numpy==1.19.3

$ python3 np.py
1.19.3
$ deactivate

Install - cluster

$ export PYENV_INSTALL=/path/to/install/dir  # suggest NOT $HOME

$ export PYENV_ROOT=${PYENV_INSTALL}/.pyenv

$ git clone https://github.com/pyenv/pyenv.git "$PYENV_ROOT"

$ git clone https://github.com/pyenv/pyenv-virtualenv.git ${PYENV_ROOT}/plugins/pyenv-virtualenv

Add the following to the end of your shell configuration file (i.e. ~/.bashrc)

export PYENV_INSTALL=/path/to/install/dir
export PYENV_ROOT=${PYENV_INSTALL}/.pyenv
# add pyenv to path
export PATH="$PYENV_ROOT/bin:$PATH"

if command -v pyenv 1>/dev/null 2>&1; then
    eval "$(pyenv init -)"
    eval "$(pyenv virtualenv-init -)"
fi

Log out and then back in

Basic Usage

$ pyenv versions
* system (set by /home/vagrant/.pyenv/version)
$ pyenv install --list
Available versions:
  2.1.3
  2.2.3
  2.3.7
  2.4.0
  2.4.1
  ...
$ pyenv install 3.9.1
Downloading Python-3.9.1.tar.xz...
...
Installed Python-3.9.1 to /home/vagrant/.pyenv/versions/3.9.1
$ pyenv versions
* system (set by /home/vagrant/.pyenv/version)
  3.9.1
$ python -V
Python 2.7.17
$ pyenv global 3.9.1  # log out and back in
$ python -V
Python 3.9.1

Real Usage

$ pyenv install 3.5.10

$ mkdir foo && cd foo && ls -la

$ python -V
3.9.1

$ pyenv local 3.5.10

$ python -V
3.5.10

$ ls -la
drwxrwxr-x 2 vagrant vagrant 4096 Jan 13 04:53 .
drwxrwxr-x 4 vagrant vagrant 4096 Jan 13 04:53 ..
-rw-rw-r-- 1 vagrant vagrant    7 Jan 13 04:53 .python-version

$ cat .python-version
3.5.10

Virtual Env Usage

$ mkdir myproject && cd myproject

$ pyenv virtualenv myproject

$ pyenv versions
system
3.5.10
* 3.9.1 (set by /home/vagrant/.pyenv/version)
3.9.1/envs/myproject
myproject

$ pyenv local myproject  # or 3.9.1/envs/myproject

$ cat .python-version
myproject

$ pyenv versions
system
3.5.10
3.9.1
3.9.1/envs/myproject
* myproject (set by /home/vagrant/tmp/myproject/.python-version)

$ pip install pyjokes

$ pyjoke
There are 10 types of people: those who understand binary and those who don't.

$ cd .. && pyjoke
pyenv: pyjoke: command not found

The `pyjoke' command exists in these Python versions:
  3.9.1/envs/myproject
  myproject

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

Example on EBI cluster

$ module load singularity/3.5.0

$ wget https://github.com/mbhall88/eipp-2019-singularity/raw/master/data/toy.bam

$ img="docker://quay.io/biocontainers/samtools:1.9--h10a08f8_12"

$ singularity -s exec "$img" samtools view -h toy.bam

singularity -s exec tells Singularity to execute a given command inside a given container (and only print errors).

"$img" specifies the container for Singularity to operate on. We will look at this component in more detail soon.

samtools view -h data/toy.bam is the command we want Singularity to execute inside the container. Notice how we can specify files that exist on our local file system?!

Retrieving a Biocontainer

tool="bwa"
tag="0.7.3a--h84994c4_4"
URI="docker://quay.io/biocontainers/${tool}:${tag}"

Reproducibility (paranoia) for Predocs What are containers and software environments and how can I become a more paranoid bioinformatician? Slides: envs-primer.netlify.app Michael Hall - Iqbal Group