Coding Guidelines

Getting Started

1. Software Carpentry can help you learn basic coding practices: working with a Unix shell, using Git, and basic Python commands. There is also a nice overview by the NSF MolSSI group on scripting for scientific computing.
2. It pays to have a reasonable familiarity with a terminal editor such as Vim, which makes editing and working on remote machines easier.
3. Python: You should familiarize yourself with PEP 8 guidelines and use tools like Pylint, Google’s yapf, or Sublime Text’s AutoPEP8 to enforce good style. It can also be helpful to use optional type hints so that others can more easily understand your code. Google has a comprehensive style guide for Python code.
4. While working in Jupyter notebooks is a great way to analyze experiments and provide tutorials of code, rigorous testing and reproducible research is more easily conducted with a well-organized and documented codebase (see here for an example layout).
5. Git: Get started with this tutorial and Chapter 2: Git Basics from the Pro Git book. Check out various git commands on its man page. For more complete information, see git documentation, particularly the user manual and the Pro Git book. For a quick interactive git tutorial, check learn git branching.
6. For reproducibility, you should always use a dependency manager like Conda or, for more complex dependencies beyond Python packages, Docker containers (see here for a short tutorial).
7. If you’re not sure what your development workflow should be, or how to use certain tools like Git, please just ask the group! We all have to learn this for the first time at some point, so there is no shame in asking. If you think “there must be a better way to do this”, there probably is.
8. We should always try to look for a general solution than an ad-hoc fix. When problems we are dealing with grow in complexity (they inevitably will), we will find solutions that are general in nature and clear in logic are easier to extend and maintain.
9. If we find ourselves doing the same thing twice, odds are that there is a better way to it. Spend 10 minutes (even hours) to learn a technique that can be implemented in 1 minute, rather than spend 1 minute to come up with a technique that takes 10 minutes. Even if the learning phase takes longer, it will usually pay off in the long run: you may likely apply the better method many times in the future and you have sharpened your coding skills.

Using Git

Git Tutorials

• This website is a useful tool for learning Git with simple, step-by-step tutorials and good visuals

Git Guidelines

1. Pull changes from the upstream repository frequently, so that conflicts can be dealt with easily. We often tend to defer this work to save time, but this debt is always paid later in much bigger proportions.
2. Merge to the upstream repository frequently (through a Pull Request), so other people can checkout your changes and stay sync.
3. Double check that changes to be submitted are correct and do not change other functionality inadvertently. Run the test suite if it exists.
4. Remove unnecessary files (any files that can be regenerated) before pushing, such as editor generated backups (*~) and temporary files (#*#), compiled executables (they are too large and should not be managed by version control systems), and simulation output from running test suites. Learn to use .gitignore files to ignore unwanted files.
5. Never commit any proprietary information (e.g., private data, passwords), even to a private repository. Triple check this before making any repository public.
6. Before merging with upstream, clean up the commit history to include only major, self-contained entries. Squash small, bookkeeping commits (e.g., test1, debug2, try3) or use the Squash and Merge functionality provided by GitLab.

Sample Git Workflow

• This brief walk-through augments, but does not replace the tutorials suggested in Point 5 of Getting Started.
• Install git on Ubuntu or WSL:

sudo apt update
sudo apt install git

• Set up your git user name and email:

git config --global user.name "Full Name"
git config --global user.email you@university.edu

• Clone a repository on GitLab from the group space to your user space.
• Clone the repository to your laptop or group server over ssh:

git clone git@gitlab.com:USERNAME/MCCCS MCCCS

This will create a new directory called “MCCCS” under the current working directory, and clone (only) the “master” branch from the remote side.

• Show the remote repositories that have been set up with your working tree:

git remote
git remote show origin

The remote repository “origin” is automatically created when you do “git clone”.

• Add the upstream repository from the group space:

git remote add upstream git@gitlab.com:group-server/MCCCS

• Examine the branches currently available:

git branch # local branches, only having "master"
git branch -r # remote branches, containing "master" and "serial"
git branch -a # show both local and remote branches

“serial” is a branch that holds some coarse graining stuff. The “master” branch is the MPI parallel version. Remote branches are a mirror image of branches in remote repositories and you NEVER switch to them directly or write to them.

• Suppose you want to work on the “serial” branch, DO NOT switch to them directly! Instead, create a corresponding local branch which will “track” the remote branch (remote-tracking branch):

git checkout -b serial origin/serial

You can now check the available branches again by “git branch -a”.

• Switch back to the “master” branch:

git checkout master

• Make some changes.
• Frequently you should commit your changes to your working tree:

git status # this will show any modified, added, and deleted files
git commit -a # commit all changes including those unstaged

This will only update your local repository.

• If changes have been made in the upstream repository, we can update our working tree by:

git fetch -p upstream
git merge upstream/master

• If you have been making changes locally that conflict with upstream, git will complain and you have to commit your changes to your working tree first. We will then use the rebase workflow to merge changes:

git rebase --onto upstream/master <LAST_COMMON_COMMIT> HEAD

where is the last commit common between upstream and your working tree. Make sure to check out the documentation and understand this command.

• Next, when we are satisfied and our changes have passed the test suites, we can push them back to origin:

git push origin master

• Create a a Pull Request on GitLab against the upstream repository.
• Now suppose we are going to do something big, making a new branch could be a good idea:

git branch test-miracle  # new branch will be based on the currently checked out branch
git checkout test-miracle  # switch to the new branch

• After the new feature has been fully tested, you can merge them back to the “master” branch. Make sure you are on one of the to-be-merged branches and merge the other one now:

git checkout master  # make sure you are on the to-be-merged branch
git merge test-miracle  # merge the other branch

If changes have been made on only “test-miracle” since the last merge, they are simply replayed on “master” (i.e., fast-forward merge). If changes have been made on both branches, they are merged intelligently (i.e., three-way merge). If there are conflicts, git merge will report them and let you resolve them, updating the rest of the tree already to the result state; you can git commit when you resolve the conflicts. If no changes conflicted, a commit is made automatically with a convenient log message (or you can do git merge --no-commit branch to review the merge result and then do the commit yourself).

• Check the differences between two files, directories, or branches:

git diff

• Browse the history of the development either in text or graphical mode:

git log  # show full history in plain text
git log serial..master  # show everything that is reachable from "master" but exclude anything that is reachable from "serial",
                        # i.e., show the changes between the current state of "master" and the point where the two branches diverge
git log serial...master  # show everything that is reachable from either one, but exclude anything that is reachable from both of them
gitk &  # show full history in graphical mode
gitk serial..master  & # same two-dot range notation as used with git log
gitk serial...master  & # same three-dot range notation as used with git log

Please refer to the user manual for specifics on these commands.

Installing Git

Mac OS X

You can install git from Fink:

fink install git

or MacPorts:

port install git-core

or brew:

brew install git

or using Git OSX Installer if you are running 10.5 or above.

Linux

On Fedora, CentOS, RedHat:

yum install git

On Ubuntu, Debian:

apt-get update
apt-get install git

Windows

Download and install the msysGit package.

Installing from source

cd ~
wget http://git-core.googlecode.com/files/git-1.7.9.tar.gz # Refer to [http://git-scm.com/download] for most current versions
tar xzf git-1.7.9.tar.gz
cd git-1.7.9
make prefix=/usr all
sudo make prefix=/usr install

Profiling Your Code

To properly speed-up your code’s implementations, it’s best to have a targeted approach. Rather than timing your code, making changes, and then retiming, it’s best to know what portions of your code is taking the most time and make improvements to that function/subroutine. This is where profilers come in. Profilers will can analyze each function of your code and determine the time it takes or the memory required to run your code.

Language Agnostic

• We can apply the Monte Carlo method for profiling our code. If we run our code and in random instances interrupt our program (ctrl-c) and determine which function our code was in, if there’s a function that’s particularly slow, we’ll find that the code is being interrupted in that section of code a majority of the time. With enough samples, we can find the sore spots in the code and make the appropriate adjustments there. For more information, see this StackOverflow post
• For cpu profiling, see VTune
• For memory profiling, see valgrind for a multi-language solution

Python

• For cpu profiling, cProfile is a built-in method for deterministically profiling your python code. After running the profiler, use the Stats class to analyze the data from the profiler. It will output the times each function is called, how long each function runs per call, the cumulative time it takes for each function to run, and much more.

• For memory usage, memray provides good visualization of how memory is allocated in your script and can simply be installed using the pip package manager.

General Tips

• When profiling your code, it may not be enough to compare a single variation of your code. Try making small changes to see what correlations you can glean (e.g. Run 100 frames of analysis and then 200 frames of analysis and see how memory/time scales. Does it scale linearly? Is this expected? Is memory that should be deallocated still allocated?)