Using IPython as a System Shell

Context switching is a developer's worst enemy. If you do a lot of interactive programming, one of the major sources of context switching is moving between your Python REPL and the system shell. IPython solves this problem by giving you access to the system shell from within the REPL itself.

Learning Outcomes

By the end of today, you'll know:

How to execute system shell commands from within IPython
What a magic function is
How to capture output from the shell as Python objects
How to work with SList objects
How to pass Python values to shell commands
When to use ! and % for executing system shell commands

Pre-requisites

Before you start today's lessons, make sure you have a working Python installation on your computer (preferably Python 3.8 or greater). You should also have IPython installed and have worked through the Day 2 lesson:

A complete primer on system shell commands is beyond the scope of this tutorial. You should be familiar with the basic commands supported by your shell. All of the commands used in this tutorial will work on macOS and most Linux systems. Many will also work in modern Windows terminals, although no guarantees are made.

IPython System Shell Access With `!`

You can execute system shell commands from within IPython by prepending them with the ! character. Everything on the line after the ! gets passed verbatim to your system shell.

For example, to see your current working directory, type !pwd into an IPython input prompt and press Enter:

In [1]: !pwd
/Users/damos

You can list the contents of your directory using !ls:

In [2]: !ls
Applications	Downloads	Music		__pycache__
Desktop		Library		Pictures	projects
Documents	Movies		Public		talkjulia

You can pass arguments and flags to shell commands as well:

In [3]: !echo "Hello, there!"
Hello, there!

Although any system command works, some do not behave as you expect:

In [4]: !cd Desktop/
/Users/damos/.local/pipx/venvs/ipython/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2468: UserWarning: You executed the system command !cd which may not work as expected. Try the IPython magic %cd instead.
  warnings.warn(

In [5]: !pwd
/Users/damos

If you try to change directories using !cd, you'll see a warning informing you that the !cd command may not work as expected. Indeed, running !pwd confirms that you haven't changed directories.

Shell commands run using ! are executed in a temporary subshell, so the effects of commands don't persist from one command to the next (unless, of course, the effect changes something on your file system, such as creating or deleting a file).

The solution is hinted at in the warning displayed when you execute !cd. Use %cd instead. Commands prepended with % are called magic functions and are one of IPython's defining features.

A Brief Introduction To Magic Functions

Magic functions, or magics for short, provide a way to use convenient names when Python syntax is not the most natural. There are numerous built-in magic functions, and you can even create your own.

You'll learn more about magic functions in a future lesson. For now, you just need to be aware that they exist. All magic functions begin with a % character, and many common system shell commands are available as magic functions.

For instance, the %cd magic function switches directories in your shell:

In [6]: %cd Desktop/
/Users/damos/Desktop

In [7]: !pwd
/Users/damos/Desktop

This time, the working directory actually changes, which you can verify using !pwd. You can also use pwd as a magic:

In [8]: %pwd
Out[8]: '/Users/damos/Desktop'

Notice that the result of %pwd isn't just printed, but captured in an output cell. This has some nice benefits since outputs captured in an output cell are stored in an output cache. You'll learn more about output caching in a future lesson.

By default, IPython lets you drop the % syntax from magic functions and just type them directly into the REPL:

In [9]: cd ~/
/Users/damos

In [10]: pwd
Out[10]: '/Users/damos'

This may be extremely convenient since it allows you to treat an IPython input prompt as if it were a shell prompt. However, if any magic function names have been used as an identifier in your Python namespace, then the magic function will no longer work without the % character:

In [11]: cd = "This overrides the cd magic function"

In [12]: cd Desktop/
  Input In [12]
    cd Desktop/
       ^
SyntaxError: invalid syntax


In [13]: %cd Desktop/   # Using %cd still works!
/Users/damos/Desktop

If you overwrite a magic function name, you can get it back by deleting the identifier with the del keyword:

In [14]: del cd

In [15]: cd ~/
/Users/damos

If you don't like using magic functions without the % character, you can turn this behavior off using the %automagic magic function:

In [16]: %automagic off

Automagic is OFF, % prefix IS needed for line magics.

In [17]: cd Desktop/
  Input In [17]
    cd Desktop/
       ^
SyntaxError: invalid syntax

Use %automagic on to turn the automagic behavior on again:

In [18]: %automagic on

Automagic is ON, % prefix IS NOT needed for line magics.

In [19]: cd Desktop/
/Users/damos/Desktop

Magic functions are a powerful tool for boosting productivity and reducing context switching inside of IPython. You'll learn more about magic functions, including how to create your own, in a later lesson.

Capturing Shell Output

You can capture output from shell commands and assign it to Python variables. This can be extremely useful during interactive sessions because it allows you to mimic the behavior of Python standard library functions for working with the filesystem without having to import anything.

For example, you can assign the output of !pwd to a variable to capture the path of your current working directory:

In [20]: path = !pwd

In [21]: path
Out[21]: ['/Users/damos/Desktop']

It's important to note that, even though you can type pwd directly into a prompt to display the current working directory, you can't use it to capture shell output:

In [22]: path = pwd
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [22], in <cell line: 1>()
----> 1 path = pwd

NameError: name 'pwd' is not defined

However, you can use %pwd to capture output:

In [23]: path = %pwd

In [24]: path
Out[24]: '/Users/damos/Desktop'

Look carefully at the result of path = !pwd and path = %pwd. In the first case, the path variable appears to point to a list. In the latter, path points to a string. This is an important distinction between using ! and % when capturing output.

Let's take a closer look at the "list" returned by !pwd:

In [25]: path = !pwd

In [26]: type(path)
Out[26]: IPython.utils.text.SList

path is not a Python list object. Instead, it is a special SList object defined by IPython. Use the ? character to see some more information:

In [27]: path?
Type:        SList
String form: ['/Users/damos/Desktop']
Length:      1
File:        ~/.local/pipx/venvs/ipython/lib/python3.10/site-packages/IPython/utils/text.py
Docstring:
List derivative with a special access attributes.

These are normal lists, but with the special attributes:

* .l (or .list) : value as list (the list itself).
* .n (or .nlstr): value as a string, joined on newlines.
* .s (or .spstr): value as a string, joined on spaces.
* .p (or .paths): list of path objects (requires path.py package)

Any values which require transformations are computed only once and
cached.

SList objects behave just like lists, except that you get some special attributes. If you want the value in the list as a single string, joined by spaces, use the .s attribute:

In [28]: path.s
Out[28]: '/Users/damos/Desktop'

If the SList object contains strings representing paths, you can use the .p attribute to get a list of path objects:

In [29]: path.p
Out[29]: [PosixPath('/Users/damos/Desktop')]

This unlocks some interesting workflows. For starters, you now have a list of Path objects, but you never had to import the pathlib module. Now you can do things like check whether or not the paths in the list are files:

In [29]: path.p[0].is_file()
Out[29]: False

SList objects are especially handy for working with directory listings, thanks to three methods:

SList.grep() returns a new SList of strings matching a regular expression
SList.fields() collects whitespace-separated fields from a string list
SList.sort() sorts an SList by a specified field

To see these in action, create an SList called paths that captures the output of ls -l:

In [30]: paths = !ls -l

In [31]: paths
Out[31]:
['total 0',
 '[email protected]  5 damos  staff   160 Mar 16 23:33 Applications',
 'drwx------+  6 damos  staff   192 Apr  4 08:53 Desktop',
 'drwx------+  6 damos  staff   192 Apr  5 09:55 Documents',
 'drwx------+  8 damos  staff   256 Apr  5 05:46 Downloads',
 '[email protected] 86 damos  staff  2752 Mar 29 20:29 Library',
 'drwx------  10 damos  staff   320 Apr  3 12:31 Movies',
 'drwx------+  4 damos  staff   128 Mar 13 23:52 Music',
 'drwx------+  4 damos  staff   128 Mar 13 22:30 Pictures',
 'drwxr-xr-x+  4 damos  staff   128 Mar 13 22:30 Public',
 'drwxr-xr-x   3 damos  staff    96 Mar 18 00:38 __pycache__',
 'drwxr-xr-x   2 damos  staff    64 Mar 14 02:17 projects',
 'drwxr-xr-x   5 damos  staff   160 Mar 17 12:37 talkjulia']

To get a list of the permissions information, use .fields(0):

In [32]: paths.fields(0)
Out[32]:
['total',
 '[email protected]',
 'drwx------+',
 'drwx------+',
 'drwx------+',
 '[email protected]',
 'drwx------',
 'drwx------+',
 'drwx------+',
 'drwxr-xr-x+',
 'drwxr-xr-x',
 'drwxr-xr-x',
 'drwxr-xr-x']

You can pass multiple field indices to .fields() as well, including negative indices:

In [33]: paths.fields(0, -1)
Out[33]:
['total 0',
 '[email protected] Applications',
 'drwx------+ Desktop',
 'drwx------+ Documents',
 'drwx------+ Downloads',
 '[email protected] Library',
 'drwx------ Movies',
 'drwx------+ Music',
 'drwx------+ Pictures',
 'drwxr-xr-x+ Public',
 'drwxr-xr-x __pycache__',
 'drwxr-xr-x projects',
 'drwxr-xr-x talkjulia']

To filter the list, use .grep() and pass to it a regular expression and a field index. For example, the following gets all paths in paths that start with a P:

In [34]: paths.grep(r"^P.*", field=-1)
Out[34]:
['drwx------+  4 damos  staff   128 Mar 13 22:30 Pictures',
 'drwxr-xr-x+  4 damos  staff   128 Mar 13 22:30 Public',
 'drwxr-xr-x   2 damos  staff    64 Mar 14 02:17 projects']

By default, regular expression in .grep() are case insensitive, which is why projects also appears in the resulting list.

You can also pass a callable to .grep():

In [35]: paths.grep(lambda x: x.startswith("P"), field=-1)
Out[35]:
['drwx------+  4 damos  staff   128 Mar 13 22:30 Pictures',
 'drwxr-xr-x+  4 damos  staff   128 Mar 13 22:30 Public']

The .sort() method can sort an SList by any field:

In [36]: paths.sort(-1)
Out[36]:
['total 0',
 '[email protected]  5 damos  staff   160 Mar 16 23:33 Applications',
 'drwx------+  6 damos  staff   192 Apr  4 08:53 Desktop',
 'drwx------+  6 damos  staff   192 Apr  5 09:55 Documents',
 'drwx------+  8 damos  staff   256 Apr  5 05:46 Downloads',
 '[email protected] 86 damos  staff  2752 Mar 29 20:29 Library',
 'drwx------  10 damos  staff   320 Apr  3 12:31 Movies',
 'drwx------+  4 damos  staff   128 Mar 13 23:52 Music',
 'drwx------+  4 damos  staff   128 Mar 13 22:30 Pictures',
 'drwxr-xr-x+  4 damos  staff   128 Mar 13 22:30 Public',
 'drwxr-xr-x   3 damos  staff    96 Mar 18 00:38 __pycache__',
 'drwxr-xr-x   2 damos  staff    64 Mar 14 02:17 projects',
 'drwxr-xr-x   5 damos  staff   160 Mar 17 12:37 talkjulia']

If the SList you are sorting only has one field, then you don't need to pass any arguments to it:

In [37]: paths = !ls

In [38]: paths.sort()
Out[38]:
['Applications',
 'Desktop',
 'Documents',
 'Downloads',
 'Library',
 'Movies',
 'Music',
 'Pictures',
 'Public',
 '__pycache__',
 'projects',
 'talkjulia']

Capturing shell output in IPython is a powerful tool to keep in your arsenal. It can greatly speed up common workflows in the terminal without having to close the REPL or switch terminal tabs or windows.

Passing Python Values To Your Shell

In addition to capturing shell output, IPython can pass values from Python objects to the shell. For example, you can pass a variable called name to the echo command:

In [39]: name = "David"

In [40]: !echo $name
David

To use a Python identifier in a shell command, either prepend a $ to it, as in the example above, or surround it with { and }:

In [41]: !echo {name}
David

This raises the question of how to use $ in a shell command, which is commonly used in bash shells to represent variables such as $HOME. In that case, use $$ to get a literal $ in the command:

In [42]: !echo $$HOME
/Users/damos

Between capturing shell output and passing Python values to your shell, IPython provides an enormously beneficial interface for working with both Python and your system all from a single environment. What workflows will this unlock for you?

Day 3 Activities

Continue to use IPython everywhere that you would use the standard Python REPL. In addition to that, try using IPython as an interface to your system shell. If you can manage it, try to use IPython only without ever leaving the REPL.

Take notes about what you like and don't like:

How was your experience using IPython as a system shell?
How can you map shell workflows into the IPython environment? How was your experience with this?
Was there anything that you struggled with or caused friction during this experience?

I hope you enjoyed today's lesson. See you tomorrow for Day 4 of Learn IPython in 10 Days.

Enjoying this course? Why not share it with a friend or colleague that you think will benefit from it? Just right-click on this link, copy the URL, and send it to them in an email!

Using IPython as a System Shell — Learn IPython Day 3