17 April 2018

648. Compiling Quantum Espresso 6.2.1 on debian 8

Time has come for me to expand my horizons and skills. As I drift farther and farther away from my origins as an organic chemist and more and more towards inorganic materials science and non-discrete, extended systems, localised orbital DFT (i.e. 'standard' DFT) is not always looking as an appropriate method.

Time thus to start using periodic systems and planewave basis sets. I've started working with nwpw and pspw in nwchem, and since I've used nwchem since ca 2002, in terms of setting up jobs and getting results it's been fairly straightforward. Understanding what all the options do and what's appropriate is another issue, but that's something I can work out empirically by comparing with benchmark systems (beginning to discover the importance of simulation cell now...)

Either way, nwchem does not yet afford a way of computing raman spectra for pspw jobs, which is one of the properties I'm interested in. Quantum Espresso, on the other hand, does, and is geared towards materials science in a way that nwchem isn't. Although quantum-espresso is included in the debian repos, I prefer to compile my own version. So, here I am.

I did this on one of my AMD nodes, but will follow up later with a post using intel MKL on my intel nodes. I only used the debian math libs, and not the AMD ACML (now replaced with ACL?), which I probably should've. Maybe I'll do an updated post on that later.

Note: I specify libblas.a instead of -lblas in the configure step since I have a custom openblas package installed as well, but which didn't work during compilation of QE.

Quantum Espresso (on an 8 core AMD processor)

'make' for gwl failed, and I didn't want to figure out why, so I skipped it. You can try doing make all instead of doing make for individual parts as I do below (type make to see what your options are).

sudo mkdir /opt/QE
sudo chown $USER /opt/QE
mkdir ~/tmp/QE -p
mkdir $HOME/scratch/QE -p
cd ~/tmp/QE
wget https://gitlab.com/QEF/q-e/-/archive/qe-6.2.1/q-e-qe-6.2.1.tar.gz
tar xvf q-e-qe-6.2.1.tar.gz
cd q-e-qe-6.2.1/

edit environment_variables
20 PREFIX=`/opt/QE` 21 BIN_DIR=$PREFIX/bin 22 PSEUDO_DIR=$PREFIX/pseudo 23 # Beware: everything in $TMP_DIR will be destroyed ! 24 TMP_DIR=$HOME/scratch/QE

sudo apt-get install libblas-dev libfftw3-dev libopenmpi-dev
./configure --prefix=/opt/QE/bin --exec-prefix=/opt/QE/bin/ FC=mpif90 BLAS_LIBS="/usr/lib/libblas/libblas.a" LIBS="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread" CPPFLAGS="-I/usr/lib/openmpi/include"
sudo apt-get install libblas-dev libfftw3-dev libopenmpi-dev
-------------------------------------------------------------------- ESPRESSO can take advantage of several optimized numerical libraries (essl, fftw, mkl...). This configure script attempts to find them, but may fail if they have been installed in non-standard locations. If a required library is not found, the local copy will be compiled. The following libraries have been found: BLAS_LIBS="/usr/lib/libblas/libblas.a" LAPACK_LIBS=-L/opt/ATLAS/lib -llapack -lblas FFT_LIBS= -lfftw3 Please check if this is what you expect. If any libraries are missing, you may specify a list of directories to search and retry, as follows: ./configure LIBDIRS="list of directories, separated by spaces" Parallel environment detected successfully.\ Configured for compilation of parallel executables. For more info, read the ESPRESSO User's Guide (Doc/users-guide.tex). --------------------------------------------------------------------

make -j8 pwall cp ld1 upf xspectra tddft
make couple epw gui gwl

echo 'export PATH=$PATH:/opt/QE/bin' >>~/.bashrc
echo 'export PSEUDO_DIR=/opt/QE/pseudo' >>~/.bashrc
cp * -R /opt/QE/

source ~/.bashrc

To benchmark:
 make test-suite

14 February 2018

647. Linda/G16 and ecce + slurm

I had the opportunity to test G16 with Linda on my cluster. Suprisingly, I observed a decent reduction in computational time for geometry optimisation, and a fair reduction for vibrational analysis.

Computational time -- geometry optimisation
1x6 cores: 100%
2x6 cores: 56% (2 nodes)
3x6 cores: 44% (3 nodes)

Computational time -- normal mode computation
1x6 cores: 100%
2x6 cores: 63% (2 nodes)
3x6 cores: 57% (3 nodes)

In no case was the speed up good enough that it actually makes long-term sense to use Linda (need less than 50% for 2 nodes and 33% or less for 3 nodes), but if you're itching for quick results, it might be worth it.

Either way, as part of this I wrote a CONFIG.Linda for ECCE and SLURM:
Gaussian-16: /opt/gaussian/g16a/g16/g16 perlPath: /usr/bin/ qmgrPath: /usr/bin/ xappsPath: /usr/bin/ Slurm { #!/bin/csh #SBATCH -p linda #SBATCH --time=$walltime #SBATCH --output=slurm.out #SBATCH --job-name=$submitFile #SBATCH --nodes=$nodes #SBATCH --cpus-per-task=$ppn } Gaussian-16FilesToRemove{ core } Gaussian-16Command{ set path = ( /opt/nbo6/bin $path ) set nnodes = `scontrol show job $SLURM_JOBID|grep NodeList|sed 's/=/\t/'|gawk '{print $2}'|tail -n 1` sed -i "2s/^/%LindaWorkers=$nnodes\n/" g16.g16in sed -i '3s/^/%UseSSH\n/' g16.g16in sed -i "s/nprocshared.*/nprocshared=$ppn./" g16.g16in setenv GAUSS_SCRDIR /home/me/scratch setenv GAUSS_EXEDIR /opt/gaussian/g16a/g16 setenv GAUSS_ARCHDIR /opt/gaussian/g16a/g16/arch setenv GAUSS_BSDDIR /opt/gaussian/g16a/g16/bsd setenv GAUSS_LEXEDIR /opt/gaussian/g16a/g16/linda-exe /opt/gaussian/g16a/g16/g16< $infile > $outfile }
The slurm partition linda has all the nodes that has g16.

26 September 2017

646. NWChem 6.6 on debian stretch (9)

I've got plenty of posts on how to compile nwchem on this blog. However, as I'm still running debian jessie on my work computers and cluster I have failed to appreciate quite how different things work in debian stretch.

This includes the compilation of ECCE (although owing to updates meant to rectify that at https://github.com/FriendsofECCE/ECCE this isn't necessarily something you'd notice) and NWChem.

Anyway, here's how to build nwchem with openmpi support under debian stretch:

1. Download source code and patches
sudo mkdir /opt/nwchem -p sudo chown $USER:$USER /opt/nwchem cd /opt/nwchem mkdir patches6.6 cd patches6.6 for i in Tddft_mxvec20.patch.gz Tools_lib64.patch.gz Config_libs66.patch.gz Cosmo_meminit.patch.gz Sym_abelian.patch.gz Xccvs98.patch.gz Dplot_tolrho.patch.gz Driver_smalleig.patch.gz Ga_argv.patch.gz Raman_displ.patch.gz Ga_defs.patch.gz Zgesvd.patch.gz Cosmo_dftprint.patch.gz Txs_gcc6.patch.gz Gcc6_optfix.patch.gz Util_gnumakefile.patch.gz Util_getppn.patch.gz Gcc6_macs_optfix.patch.gz Xatom_vdw.patch.gz Notdir_fc.patch.gz Hfmke.patch.gz Cdfti2nw66.patch.gz ; do wget http://www.nwchem-sw.org/download.php?f=$i ; done gunzip *.gz cat *.patch > consolidated.patch cd .. wget http://www.nwchem-sw.org/download.php?f=Nwchem-6.6.revision27746-src.2015-10-20.tar.gz -O Nwchem-6.6.revision27746-src.2015-10-20.tar.gz tar xvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz

Apply patches:
cd /opt/nwchem/nwchem-6.6/ cp /opt/nwchem/patches6.6/consolidated.patch . patch -p0 < consolidated.patch
In addition, you can do the following to print more information so that GabEdit properly shows MOs:
cd nwchem-6.6/ grep -rl "do klo = 0, min(n-1,9), 2"|xargs -I {} sed -i 's/do klo = 0, min(n-1,9), 2/do klo = 0, min(n-1,199), 2/g' {} grep -rl " 0.15d0,"|grep -v "atomdata"|xargs -I {} sed -i 's/0.15d0/0.01d0/g' {}

2. Install dependencies and compile.
 How exactly it is done depends on your compiler and math libs. I've used red for examples with INTEL mkl, blue for AMD acml and orange for the free openblas. ifort is shown in purple and gfortran like this.You should only use one compiler + one math lib.
Hint: also make sure your ld.conf.so.d files contain paths to your math libs.
cd /opt/nwchem/nwchem-6.6 sudo apt-get install build-essential gfortran python2.7-dev libopenmpi-dev openmpi-bin export NWCHEM_TOP=`pwd` export LARGE_FILES=TRUE export TCGRSH=/usr/bin/ssh export NWCHEM_TOP=`pwd` export NWCHEM_TARGET=LINUX64 export NWCHEM_MODULES="all" export PYTHONVERSION=2.7 export PYTHONHOME=/usr export BLASOPT="-L/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/ -lmkl_core -lmkl_sequential -lmkl_intel_ilp64" export LIBRARY_PATH="$LIBRARY_PATH:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/" export BLASOPT="-L/opt/acml/acml5.3.1/gfortran64_int64/lib -lacml" export LIBRARY_PATH="$LIBRARY_PATH:/opt/acml/acml5.3.1/gfortran64_int64/lib" export BLASOPT="-L/opt/openblas/lib -lopenblas" export LIBRARY_PATH="$LIBRARY_PATH:/opt/openblas/lib" export USE_MPI=y export USE_MPIF=y export USE_MPIF4=y export ARMCI_NETWORK=SOCKETS cd $NWCHEM_TOP/src make clean make nwchem_config make FC=ifort 1> make.log 2>make.err make FC=gfortran 1> make.log 2>make.err cd $NWCHEM_TOP/contrib export FC=ifort export FC=gfortran ./getmem.nwchem

3. Run
mpirun -n 8 /opt/nwchem-6.6/bin/LINUX64/nwchem input.nw > output.nw