Python Virtual Environment

A virtual environment is a tool that helps you keep dependencies required by different projects separate by creating isolated Python environments for them. This means that each project can have its own dependencies, regardless of what dependencies every other project has.

Why do you need a virtual environment?

Imagine a scenario where you are working on two Machine Learning projects both based on TensorFlow fo Python. One project uses TensorFlow 1.5 while the other one uses TensorFlow 2.1. In such cases, a virtual environment can be really useful to maintain dependencies of both projects.

When and where to use a virtual environment?

By default, every project on your system will use these same directories to store and retrieve site packages (third party libraries). How does this matter? Now, in the previous example of two web applications you have two versions of TensorFlow. This is a real problem for Python since it can’t differentiate between versions in the site-packages directory. So both v1.5 and v2.1 would reside in the same directory with the same name. This is where virtual environments come into play. To solve this problem, we just need to create two separate virtual environments for both projects.The great thing about this is that there are no limits to the number of environments you can have since they’re just directories containing a few scripts.

Virtual Environments should be used whenever you work on any Python based project. It is generally good to have one new virtual environment for every Python based project you work on. So the dependencies of every project are isolated from the system and each other.

Note that we only install Python 3. We highly recommend you upgrade your Python scripts to version 3 since version 2 came to EOL (end of life) on January 2020.

PIP

pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

Installing packages

pip supports installing from PyPI, version control, local projects, and directly from distribution files.

Install latest version

$ pip install SomePackage

Install specific version

$ pip install SomePackage==1.0.4

Install minimum version

$ pip install 'SomePackage>=1.0.4

Installing packages listed in the requirements file

$ pip install -r requirements.txt

Replicate an environment with all the installed packages

$ pip freeze > requirements.txt

This will create a requirements.txt file, which contains a simple list of all the packages in the current environment, and their respective versions. You can see the list of installed packages without the requirements format using pip list. Later it will be easier for a different developer (or you, if you need to re-create the environment) to install the same packages using the same versions:

$ pip install -r requirements.txt

This can help ensure consistency across installations, across deployments, and across developers.

Pip install requires root access, to install a package without root access, use “--user” option

$ pip install –user SomePackage

Upgrade a package

$ pip install –upgrade SomePackage

Uninstalling packages

$ pip uninstall SomePackage

pip also performs an automatic uninstall of an old version of a package before upgrading to a newer version.

List installed packages

$ pip list

Show installed package details

$ pip show SomePackage

Search for packages

$ pip search “query”

The query will be used to search the names and summaries of all packages.

Virtualenv

virtualenv is a tool to create isolated Python environments.

virtualenv creates a folder which contains all the necessary executables to use the packages that a Python project would need.

Installation

There are a number of ways to install virtualenv on your system.

$ sudo apt install python-virtualenv

$ sudo easy_install virtualenv

$ sudo pip install virtualenv

Basic usage

Create a virtual environment for a project

$ cd project_folder

$ virtualenv venv

virtualenv venv will create a folder in the current directory which will contain the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual name will place the files in the current directory instead.

This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv.

You can also use the Python interpreter of your choice (like python3.5).

$ virtualenv -p python3.5 venv

or change the interpreter globally with an env variable in ~/.bashrc:

$ export VIRTUALENVWRAPPER\_PYTHON=/usr/bin/python3.5

To begin using the virtual environment, it needs to be activated

$ source venv/bin/activate

  • The name of the current virtual environment will now appear on the left of the prompt (e.g. (venv)Your-Computer:project_folder UserName$) to let you know that it’s active. From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

Install packages using the pip command

$ pip install SomePackage

If you are done working in the virtual environment for the moment, you can deactivate it

$ deactivate

  • This puts you back to the system’s default Python interpreter with all its installed libraries.

Delete a virtual environment

Delete its folder:

$ rm -rf venv

Conda

Conda is an open source package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. It works on Linux, macOS and Windows, and was created for Python programs but can package and distribute any software.

Installation

Managing conda

Verify that conda is installed and running on your system

$ conda –version

Update conda to the current version

$ conda update conda

Managing environments

Conda allows you to create separate environments containing files, packages and their dependencies that will not interact with other environments.

When you begin using conda, you already have a default environment named base.

Create a new environment

$ conda create –name EnvironmentName PackageName

Activate an environment

$ conda activate EnvironmentName

  • For conda versions prior to 4.6:

$ source activate EnvironmentName

  • Now that you are in your environment, any conda commands you type will go to that environment until you deactivate it.

Deactivate an environment

$ conda deactivate

List of all your environments

$ conda info –envs

$ conda environments

base /home/username/Anaconda3
 
snowflakes \* /home/username/Anaconda3/envs/snowflakes
  • The active environment is the one with an asterisk (*).

Change your current environment back to the default (base)

$ conda activate

  • For versions prior to conda 4.6:

$ source activate

Remove an environment

$ conda remove –name EnvironmentName –all

Managing packages

Search for packages

$ conda search scipy

Install a package

$ conda install scipy

List installed packages

$ conda list

We install pipvirtualenv and conda on Remote & Physical workstations.

We deliver Conda through miniconda3 on the scientific application stack.

Working with virtual environment in a GitLab project

1. Create your virtual environment in your project directory

$ virtualenv venv

2. Add your venv directory to .gitignore

$ echo 'venv' > .gitignore

  • This keeps your virtual environment out of source control.

3. Activate the environment

$ source venv/bin/activate

4. Install the packages

$ pip install SomePackage

5. Freeze the requirements

$ pip freeze > requirements.txt

6. Check requirements.txt into source control

$ git add requirements.txt

The basic idea here is simple. Your virtual environment stays in its own folder, which you don’t check into source control. That way your environment does not pollute your project’s root. The requirements, on the other hand, do get checked into source control.

References