Context switching is a developer's worst enemy. If you do a lot of interactive programming, one of the major sources of context switching is moving between your Python REPL and the system shell. IPython solves this problem by giving you access to the system shell from within the REPL itself.
Learning Outcomes
By the end of today, you'll know:
- How to execute system shell commands from within IPython
- What a magic function is
- How to capture output from the shell as Python objects
- How to work with
SListobjects - How to pass Python values to shell commands
- When to use
!and%for executing system shell commands
Pre-requisites
Before you start today's lessons, make sure you have a working Python installation on your computer (preferably Python 3.8 or greater). You should also have IPython installed and have worked through the Day 2 lesson:
David Amos
A complete primer on system shell commands is beyond the scope of this tutorial. You should be familiar with the basic commands supported by your shell. All of the commands used in this tutorial will work on macOS and most Linux systems. Many will also work in modern Windows terminals, although no guarantees are made.
IPython System Shell Access With !
You can execute system shell commands from within IPython by prepending them with the ! character. Everything on the line after the ! gets passed verbatim to your system shell.
For example, to see your current working directory, type !pwd into an IPython input prompt and press Enter:
In [1]: !pwd
/Users/damosYou can list the contents of your directory using !ls:
In [2]: !ls
Applications Downloads Music __pycache__
Desktop Library Pictures projects
Documents Movies Public talkjuliaYou can pass arguments and flags to shell commands as well:
In [3]: !echo "Hello, there!"
Hello, there!Although any system command works, some do not behave as you expect:
In [4]: !cd Desktop/
/Users/damos/.local/pipx/venvs/ipython/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2468: UserWarning: You executed the system command !cd which may not work as expected. Try the IPython magic %cd instead.
warnings.warn(
In [5]: !pwd
/Users/damosIf you try to change directories using !cd, you'll see a warning informing you that the !cd command may not work as expected. Indeed, running !pwd confirms that you haven't changed directories.
Shell commands run using ! are executed in a temporary subshell, so the effects of commands don't persist from one command to the next (unless, of course, the effect changes something on your file system, such as creating or deleting a file).
The solution is hinted at in the warning displayed when you execute !cd. Use %cd instead. Commands prepended with % are called magic functions and are one of IPython's defining features.
A Brief Introduction To Magic Functions
Magic functions, or magics for short, provide a way to use convenient names when Python syntax is not the most natural. There are numerous built-in magic functions, and you can even create your own.
You'll learn more about magic functions in a future lesson. For now, you just need to be aware that they exist. All magic functions begin with a % character, and many common system shell commands are available as magic functions.
For instance, the %cd magic function switches directories in your shell:
In [6]: %cd Desktop/
/Users/damos/Desktop
In [7]: !pwd
/Users/damos/DesktopThis time, the working directory actually changes, which you can verify using !pwd. You can also use pwd as a magic:
In [8]: %pwd
Out[8]: '/Users/damos/Desktop'Notice that the result of %pwd isn't just printed, but captured in an output cell. This has some nice benefits since outputs captured in an output cell are stored in an output cache. You'll learn more about output caching in a future lesson.
By default, IPython lets you drop the % syntax from magic functions and just type them directly into the REPL:
In [9]: cd ~/
/Users/damos
In [10]: pwd
Out[10]: '/Users/damos'This may be extremely convenient since it allows you to treat an IPython input prompt as if it were a shell prompt. However, if any magic function names have been used as an identifier in your Python namespace, then the magic function will no longer work without the % character:
In [11]: cd = "This overrides the cd magic function"
In [12]: cd Desktop/
Input In [12]
cd Desktop/
^
SyntaxError: invalid syntax
In [13]: %cd Desktop/ # Using %cd still works!
/Users/damos/DesktopIf you overwrite a magic function name, you can get it back by deleting the identifier with the del keyword:
In [14]: del cd
In [15]: cd ~/
/Users/damosIf you don't like using magic functions without the % character, you can turn this behavior off using the %automagic magic function:
In [16]: %automagic off
Automagic is OFF, % prefix IS needed for line magics.
In [17]: cd Desktop/
Input In [17]
cd Desktop/
^
SyntaxError: invalid syntaxUse %automagic on to turn the automagic behavior on again:
In [18]: %automagic on
Automagic is ON, % prefix IS NOT needed for line magics.
In [19]: cd Desktop/
/Users/damos/DesktopMagic functions are a powerful tool for boosting productivity and reducing context switching inside of IPython. You'll learn more about magic functions, including how to create your own, in a later lesson.
Capturing Shell Output
You can capture output from shell commands and assign it to Python variables. This can be extremely useful during interactive sessions because it allows you to mimic the behavior of Python standard library functions for working with the filesystem without having to import anything.
For example, you can assign the output of !pwd to a variable to capture the path of your current working directory:
In [20]: path = !pwd
In [21]: path
Out[21]: ['/Users/damos/Desktop']It's important to note that, even though you can type pwd directly into a prompt to display the current working directory, you can't use it to capture shell output:
In [22]: path = pwd
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [22], in <cell line: 1>()
----> 1 path = pwd
NameError: name 'pwd' is not definedHowever, you can use %pwd to capture output:
In [23]: path = %pwd
In [24]: path
Out[24]: '/Users/damos/Desktop'Look carefully at the result of path = !pwd and path = %pwd. In the first case, the path variable appears to point to a list. In the latter, path points to a string. This is an important distinction between using ! and % when capturing output.
Let's take a closer look at the "list" returned by !pwd:
In [25]: path = !pwd
In [26]: type(path)
Out[26]: IPython.utils.text.SListpath is not a Python list object. Instead, it is a special SList object defined by IPython. Use the ? character to see some more information:
In [27]: path?
Type: SList
String form: ['/Users/damos/Desktop']
Length: 1
File: ~/.local/pipx/venvs/ipython/lib/python3.10/site-packages/IPython/utils/text.py
Docstring:
List derivative with a special access attributes.
These are normal lists, but with the special attributes:
* .l (or .list) : value as list (the list itself).
* .n (or .nlstr): value as a string, joined on newlines.
* .s (or .spstr): value as a string, joined on spaces.
* .p (or .paths): list of path objects (requires path.py package)
Any values which require transformations are computed only once and
cached.SList objects behave just like lists, except that you get some special attributes. If you want the value in the list as a single string, joined by spaces, use the .s attribute:
In [28]: path.s
Out[28]: '/Users/damos/Desktop'If the SList object contains strings representing paths, you can use the .p attribute to get a list of path objects:
In [29]: path.p
Out[29]: [PosixPath('/Users/damos/Desktop')]This unlocks some interesting workflows. For starters, you now have a list of Path objects, but you never had to import the pathlib module. Now you can do things like check whether or not the paths in the list are files:
In [29]: path.p[0].is_file()
Out[29]: FalseSList objects are especially handy for working with directory listings, thanks to three methods:
SList.grep()returns a newSListof strings matching a regular expressionSList.fields()collects whitespace-separated fields from a string listSList.sort()sorts anSListby a specified field
To see these in action, create an SList called paths that captures the output of ls -l:
In [30]: paths = !ls -l
In [31]: paths
Out[31]:
['total 0',
'[email protected] 5 damos staff 160 Mar 16 23:33 Applications',
'drwx------+ 6 damos staff 192 Apr 4 08:53 Desktop',
'drwx------+ 6 damos staff 192 Apr 5 09:55 Documents',
'drwx------+ 8 damos staff 256 Apr 5 05:46 Downloads',
'[email protected] 86 damos staff 2752 Mar 29 20:29 Library',
'drwx------ 10 damos staff 320 Apr 3 12:31 Movies',
'drwx------+ 4 damos staff 128 Mar 13 23:52 Music',
'drwx------+ 4 damos staff 128 Mar 13 22:30 Pictures',
'drwxr-xr-x+ 4 damos staff 128 Mar 13 22:30 Public',
'drwxr-xr-x 3 damos staff 96 Mar 18 00:38 __pycache__',
'drwxr-xr-x 2 damos staff 64 Mar 14 02:17 projects',
'drwxr-xr-x 5 damos staff 160 Mar 17 12:37 talkjulia']To get a list of the permissions information, use .fields(0):
In [32]: paths.fields(0)
Out[32]:
['total',
'[email protected]',
'drwx------+',
'drwx------+',
'drwx------+',
'[email protected]',
'drwx------',
'drwx------+',
'drwx------+',
'drwxr-xr-x+',
'drwxr-xr-x',
'drwxr-xr-x',
'drwxr-xr-x']You can pass multiple field indices to .fields() as well, including negative indices:
In [33]: paths.fields(0, -1)
Out[33]:
['total 0',
'[email protected] Applications',
'drwx------+ Desktop',
'drwx------+ Documents',
'drwx------+ Downloads',
'[email protected] Library',
'drwx------ Movies',
'drwx------+ Music',
'drwx------+ Pictures',
'drwxr-xr-x+ Public',
'drwxr-xr-x __pycache__',
'drwxr-xr-x projects',
'drwxr-xr-x talkjulia']To filter the list, use .grep() and pass to it a regular expression and a field index. For example, the following gets all paths in paths that start with a P:
In [34]: paths.grep(r"^P.*", field=-1)
Out[34]:
['drwx------+ 4 damos staff 128 Mar 13 22:30 Pictures',
'drwxr-xr-x+ 4 damos staff 128 Mar 13 22:30 Public',
'drwxr-xr-x 2 damos staff 64 Mar 14 02:17 projects']By default, regular expression in .grep() are case insensitive, which is why projects also appears in the resulting list.
You can also pass a callable to .grep():
In [35]: paths.grep(lambda x: x.startswith("P"), field=-1)
Out[35]:
['drwx------+ 4 damos staff 128 Mar 13 22:30 Pictures',
'drwxr-xr-x+ 4 damos staff 128 Mar 13 22:30 Public']The .sort() method can sort an SList by any field:
In [36]: paths.sort(-1)
Out[36]:
['total 0',
'[email protected] 5 damos staff 160 Mar 16 23:33 Applications',
'drwx------+ 6 damos staff 192 Apr 4 08:53 Desktop',
'drwx------+ 6 damos staff 192 Apr 5 09:55 Documents',
'drwx------+ 8 damos staff 256 Apr 5 05:46 Downloads',
'[email protected] 86 damos staff 2752 Mar 29 20:29 Library',
'drwx------ 10 damos staff 320 Apr 3 12:31 Movies',
'drwx------+ 4 damos staff 128 Mar 13 23:52 Music',
'drwx------+ 4 damos staff 128 Mar 13 22:30 Pictures',
'drwxr-xr-x+ 4 damos staff 128 Mar 13 22:30 Public',
'drwxr-xr-x 3 damos staff 96 Mar 18 00:38 __pycache__',
'drwxr-xr-x 2 damos staff 64 Mar 14 02:17 projects',
'drwxr-xr-x 5 damos staff 160 Mar 17 12:37 talkjulia']If the SList you are sorting only has one field, then you don't need to pass any arguments to it:
In [37]: paths = !ls
In [38]: paths.sort()
Out[38]:
['Applications',
'Desktop',
'Documents',
'Downloads',
'Library',
'Movies',
'Music',
'Pictures',
'Public',
'__pycache__',
'projects',
'talkjulia']Capturing shell output in IPython is a powerful tool to keep in your arsenal. It can greatly speed up common workflows in the terminal without having to close the REPL or switch terminal tabs or windows.
Passing Python Values To Your Shell
In addition to capturing shell output, IPython can pass values from Python objects to the shell. For example, you can pass a variable called name to the echo command:
In [39]: name = "David"
In [40]: !echo $name
DavidTo use a Python identifier in a shell command, either prepend a $ to it, as in the example above, or surround it with { and }:
In [41]: !echo {name}
DavidThis raises the question of how to use $ in a shell command, which is commonly used in bash shells to represent variables such as $HOME. In that case, use $$ to get a literal $ in the command:
In [42]: !echo $$HOME
/Users/damosBetween capturing shell output and passing Python values to your shell, IPython provides an enormously beneficial interface for working with both Python and your system all from a single environment. What workflows will this unlock for you?
Day 3 Activities
Continue to use IPython everywhere that you would use the standard Python REPL. In addition to that, try using IPython as an interface to your system shell. If you can manage it, try to use IPython only without ever leaving the REPL.
Take notes about what you like and don't like:
- How was your experience using IPython as a system shell?
- How can you map shell workflows into the IPython environment? How was your experience with this?
- Was there anything that you struggled with or caused friction during this experience?
I hope you enjoyed today's lesson. See you tomorrow for Day 4 of Learn IPython in 10 Days.
Enjoying this course? Why not share it with a friend or colleague that you think will benefit from it? Just right-click on this link, copy the URL, and send it to them in an email!