Python


We have a large number (~600) of python modules already installed:

benji@seraph ~ $ module load python
benji@seraph ~ $ python -i
Python 2.7.10 (default, Mar  6 2017, 15:11:24) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')

BaseHTTPServer      calendar_resource   linuxaudiodev       select
Bastion             cgi                 llvmlite            sets
Bio                 cgitb               locale              setuptools
BioSQL              chardet             logging             sgmllib
CDROM               cherrypy            macpath             sha
CGIHTTPServer       chunk               macurl2path         shelve
Canvas              click               mailbox             shlex
ConfigParser        cmath               mailcap             shutil
Cookie              cmd                 marglikehood        signal
Crypto              code                markupbase          simplegeneric
Cython              codecs              markupsafe          singledispatch
DLFCN               codeop              marshal             singledispatch_helpers
Dialog              codesearch          math                sip
DocXMLRPCServer     cogent              matplotlib          sipconfig
FileDialog          collections         md5                 sipdistutils
FixTk               colorsys            media               site
HTMLParser          commands            mhlib               sites
IN                  compileall          mimetools           six
IPython             compiler            mimetypes           skbio
MimeWriter          concurrent          mimify              sklearn
PyQt5               contacts            mistune             smtpd
Queue               contentforshopping  mmap                smtplib
ScrolledText        contextlib          mock                sndhdr
SimpleDialog        cookielib           modulefinder        socket
SimpleHTTPServer    copy                mpi4py              socketserver
SimpleXMLRPCServer  copy_reg            multifile           sphinx
SocketServer        copyreg             multiprocessing     spreadsheet
StringIO            crypt               mutex               spreadsheets
TYPES               csv                 natsort             spwd
Tix                 ctypes              nbconvert           spyder
Tkconstants         curses              nbformat            spyder_breakpoints
Tkdnd               cycler              netrc               spyder_io_dcm
Tkinter             cython              networkx            spyder_io_hdf5
UserDict            cythonmagic         new                 spyder_profiler
UserList            datetime            nis                 spyder_pylint
UserString          dateutil            nntplib             sqlite3
_LWPCookieJar       dbhash              nose                sre
_MozillaCookieJar   decimal             notebook            sre_compile
__builtin__         decorator           ntpath              sre_constants
__future__          difflib             nturl2path          sre_parse
_abcoll             dill                numba               ssl
_ast                dircache            numbers             stat
_bisect             dis                 numexpr             statvfs
_codecs             distutils           numpy               storemagic
_codecs_cn          django              oauth               string
_codecs_hk          docs                opcode              stringold
_codecs_iso2022     doctest             opensearch          stringprep
_codecs_jp          docutils            operator            strop
_codecs_kr          dublincore          optparse            struct
_codecs_tw          dumbdbm             os                  subprocess
_collections        dummy_thread        os2emxpath          sumac
_csv                dummy_threading     ossaudiodev         sunau
_ctypes             easy_install        pandas              sunaudio
_ctypes_test        email               parse_bed           symbol
_dummy_thread       emperor             parse_str           sympyprinting
_elementtree        encodings           parser              symtable
_functools          ensurepip           past                sys
_hashlib            enum                path                sysconfig
_heapq              errno               pbr                 syslog
_hotshot            ete2                pdb                 tables
_io                 exceptions          pep8                tabnanny
_json               exif                pexpect             tarfile
_locale             external            photos              telnetlib
_lsprof             fastStructure       pickle              tempfile
_markupbase         fcntl               pickleshare         tensorflow
_multibytecodec     filecmp             pickletools         termios
_multiprocessing    fileinput           pip                 test
_osx_support        finance             pipes               test_path
_pyio               fnmatch             pkg_resources       tests
_random             formatter           pkgutil             textwrap
_socket             fpformat            platform            this
_sqlite3            fractions           plistlib            thread
_sre                ftplib              popen2              threading
_ssl                funcsigs            poplib              time
_strptime           functools           posix               timeit
_struct             functools32         posixfile           tkColorChooser
_symtable           future              posixpath           tkCommonDialog
_sysconfigdata      future_builtins     pprint              tkFileDialog
_testcapi           gc                  profile             tkFont
_thread             gdata               projecthosting      tkMessageBox
_threading_local    genericpath         pstats              tkSimpleDialog
_warnings           geo                 psutil              tkinter
_weakref            getopt              pty                 tlslite
_weakrefset         getpass             ptyprocess          toaiff
abc                 gettext             pwd                 token
acl                 glob                py_compile          tokenize
admixprop           grp                 pyclbr              tornado
aifc                gzip                pydoc               toyplot
allelefreq          h5py                pydoc_data          trace
alt                 hashlib             pyexpat             traceback
analytics           health              pyflakes            traitlets
antigravity         heapq               pygments            ttk
anvio               hmac                pylab               tty
anydbm              hotshot             pylint              turtle
apps                html                pynast              types
argparse            htmlentitydefs      pyparsing           unicodedata
array               htmllib             pyqi                unittest
ast                 http                pyrad               urllib
asynchat            httplib             pysam               urllib2
asyncore            idlelib             pytz                urlparse
atexit              ihooks              pyximport           user
atom                imagesize           qcli                utils
audiodev            imaplib             qiime               uu
audioop             imghdr              qiime_default_reference uuid
autoreload          imp                 qiime_test_data     virtualenv
babel               importlib           qtawesome           virtualenv_support
backports           imputil             qtconsole           warnings
backports_abc       inspect             qtpy                wave
base64              io                  queue               weakref
bdb                 ipykernel           quopri              webbrowser
bfillings           ipyparallel         random              webmastertools
binascii            ipyrad              re                  wheel
binhex              ipython_genutils    readline            whichdb
biom                itertools           repr                winreg
bisect              jedi                reprlib             wsgiref
blogger             jinja2              requests            xdrlib
books               json                resource            xml
bottle              jsonschema          rexec               xmllib
bsddb               jupyter             rfc822              xmlrpc
builtins            jupyter_client      rlcompleter         xmlrpclib
burrito             jupyter_core        rmagic              xxsubtype
bz2                 keyword             robotparser         youtube
cPickle             lib2to3             rope                zipfile
cProfile            libfuturize         runpy               zipimport
cStringIO           libpasteurize       sched               zlib
calendar            linecache           scipy               zmq

Enter any module name to get more help.  Or, type "modules spam" to search
for modules whose descriptions contain the word "spam".

>>>

But if you find yourself missing something you can -

Install a Python module using a Virtual Environment

First load the module and create a virtualenv (you only need to do this step once)

benji@acheron ~ $ module load python
benji@acheron ~ $ virtualenv --system-site-packages myvenv
New python executable in myvenv/bin/python2
Also creating executable in myvenv/bin/python
Installing setuptools............done.
Installing pip...............done.

Now activate the virtualenv

benji@acheron ~ $ source myvenv/bin/activate

Install a python package using pip, in this case taxtastic

(myvenv)benji@acheron ~ $ pip install taxtastic
Downloading/unpacking taxtastic
  Downloading taxtastic-0.5.4.tar.gz (75kB): 75kB downloaded
  Running setup.py egg_info for package taxtastic

Downloading/unpacking sqlalchemy>=0.7 (from taxtastic)
  Downloading SQLAlchemy-1.0.9.tar.gz (4.7MB): 4.7MB downloaded
  Running setup.py egg_info for package sqlalchemy

    warning: no files found matching '*.jpg' under directory 'doc'
    warning: no files found matching '*.mako' under directory 'doc'
    warning: no files found matching 'distribute_setup.py'
    warning: no files found matching 'sa2to3.py'
    warning: no files found matching 'ez_setup.py'
    no previously-included directories found matching 'doc/build/output'
Downloading/unpacking decorator (from taxtastic)
  Downloading decorator-4.0.4.tar.gz (67kB): 67kB downloaded
  Running setup.py egg_info for package decorator

Requirement already satisfied (use --upgrade to upgrade): biopython in /opt/modules/devel/python/2.7.5/lib/python2.7/site-packages (from taxtastic)
Downloading/unpacking xlrd (from taxtastic)
  Downloading xlrd-0.9.4.tar.gz (322kB): 322kB downloaded
  Running setup.py egg_info for package xlrd

Installing collected packages: taxtastic, sqlalchemy, decorator, xlrd
  Running setup.py install for taxtastic
    got version from file taxtastic/_version.py {'version': '0.5.4', 'full': '917abacee145bef7506478e005827ef1bb2a9e27'}
    changing mode of build/scripts-2.7/taxit from 644 to 755
    UPDATING build/lib/taxtastic/_version.py

    changing mode of /mnt/lfs/benji/myvenv/bin/taxit to 755
  Running setup.py install for sqlalchemy
    building 'sqlalchemy.cprocessors' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/modules/devel/python/2.7.5/include/python2.7 -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/processors.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/processors.o -o build/lib.linux-x86_64-2.7/sqlalchemy/cprocessors.so
    building 'sqlalchemy.cresultproxy' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/modules/devel/python/2.7.5/include/python2.7 -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/resultproxy.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/resultproxy.o -o build/lib.linux-x86_64-2.7/sqlalchemy/cresultproxy.so
    building 'sqlalchemy.cutils' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/modules/devel/python/2.7.5/include/python2.7 -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/utils.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/utils.o -o build/lib.linux-x86_64-2.7/sqlalchemy/cutils.so

    warning: no files found matching '*.jpg' under directory 'doc'
    warning: no files found matching '*.mako' under directory 'doc'
    warning: no files found matching 'distribute_setup.py'
    warning: no files found matching 'sa2to3.py'
    warning: no files found matching 'ez_setup.py'
    no previously-included directories found matching 'doc/build/output'
  Running setup.py install for decorator

  Running setup.py install for xlrd
    changing mode of build/scripts-2.7/runxlrd.py from 644 to 755

    changing mode of /mnt/lfs/benji/myvenv/bin/runxlrd.py to 755
Successfully installed taxtastic sqlalchemy decorator xlrd
Cleaning up...

See if we can use the module

(myvenv)benji@acheron ~ $ which taxit
/mnt/lfs/benji/myvenv/bin/taxit
(myvenv)benji@acheron ~ $ taxit -h
usage: taxit [-h] [-V]

             {help,add_nodes,add_to_taxtable,check,composition,create,findcompany,info,lonelynodes,merge,new_database,refpkg_intersection,reroot,rollback,rollforward,rp,strip,taxids,taxtable,update,update_taxids}
             ...

Creation, validation, and modification of reference packages for use with
`pplacer` and related software.

positional arguments:
  {help,add_nodes,add_to_taxtable,check,composition,create,findcompany,info,lonelynodes,merge,new_database,refpkg_intersection,reroot,rollback,rollforward,rp,strip,taxids,taxtable,update,update_taxids}
    help                Detailed help for actions using `help <action>`
    add_nodes           Add new nodes to a database containing a taxonomy.
    add_to_taxtable     Add nodes to a taxtable
    check               Validate a reference package.
    composition         Show taxonomic composition of a reference package.
    create              Create a reference package
    findcompany         Find company for lonely nodes.
    info                Show information about reference packages.
    lonelynodes         Extracts tax ids of all lonely nodes in a taxtable.
    merge               Identify merged taxids and provide replacements.
    new_database        Create a database containing an entire taxonomy
    refpkg_intersection
                        Find the intersection of a taxtable and a refpkg's
                        taxonomy.
    reroot              Taxonomically reroots a reference package
    rollback            Undo an operation performed on a refpkg.
    rollforward         Restore a change to a refpkg immediately after being
                        reverted.
    rp                  Resolve path; get the path to a file in the reference
                        package.
    strip               Remove rollback and rollforward information from a
                        refpkg
    taxids              Convert a list of taxonomic names into a list of
                        tax_ids
    taxtable            Create a tabular representation of taxonomic lineages
    update              Add or modify files or metadata in a refpkg
    update_taxids       Update obsolete tax_ids

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         Print the version number and exit
(myvenv)benji@acheron ~ $

Success. When we’re done, just deactivate it.

(myvenv)benji@acheron ~ $ deactivate
benji@acheron ~ $

More info regarding Virtual Environments here

JupyterHub

Several of our standalone servers run JupyterHub, an GUI environment for Python based on Jupyter Notebooks. For example: Visit https://crick.ibest.uidaho.edu:8000