IPython's magic function system exposes a rich set of commands that can be used to configure IPython, run and edit code, and inspect code objects. Magic functions, sometimes referred to as magic commands or magics, are one of IPython's defining features.

Learning Outcomes

By the end of today, you'll know:

  • What line magics are
  • What cell magics are
  • View the IPython quick reference card using the %quickref magic command
  • How to explore code objects using the %pdef, %pdoc, %pfile magic commands
  • How to view interactive variables with %who and %whos
  • How to configure IPython using the %color and %configure magic commands
  • How to install packages using the %pip magic command

Pre-requisites

Before you start today's lessons, make sure you have a working Python installation on your computer (preferably Python 3.8 or greater). You should also have IPython installed and have worked through the Day 3 lesson:

Using IPython as a System Shell
If you do a lot of interactive programming, one of the major sources of context switching is moving between your Python REPL and the system shell. IPython solves this problem by giving you access to the system shell from within the REPL itself.
David AmosDavid Amos

IPython Magic Commands

Magic commands are, in a lot of ways, the bread and butter of IPython. Without them, IPython is just a fancy Python REPL with colors and a few additional features like better tab completion and multiline editing.

There are two types of magic commands:

  1. Line magics start with a % character and have a system command style syntax. They operate on a single line and arguments are passed to the command without any parentheses or quotes. Line magics can return values and can be used on the right-hand side of an assignment expression.
  2. Cell magics start with %% and operate on an entire IPython cell.

This tutorial focuses exclusively on line magics.

In yesterday's lesson, you learned about magic commands like %cd and %ls for working with the system shell, but there are so many more at your fingertips. A full list of available magic commands can be found in the IPython docs.

Today, you'll learn about magic commands that can be used to explore code objects, configure the IPython REPL, and install packages with pip. Future lessons will introduce even more magic commands relevant to different topics, such as running, editing, timing, and profiling code.

At any time, you can use the %magic magic command to bring up the docs for the magic function system. Just type %magic into an input prompt and press Enter and you'll see the following screen:

Use the PgUp, PgDn, and arrow keys to navigate the help screen. When you're done, press q to return to the REPL.

The IPython Quick Reference Card

IPython is a powerful Python REPL, and there's a lot of information to keep up with. Wouldn't it be great to have a cheat sheet available to you at any time? Fortunately, there is! And it's available directly within the REPL any time you need it.

Type %quickref into an input prompt and press enter to see the quick reference sheet:

As usual, use PgUp, PgDn, and the arrow keys to navigate the quick reference card. When you're done, press q to return to the REPL.

Thanks to IPython's automagic functionality, which you learned about on Day 3, you don't even need to type the % character. That is, the following command works just fine:

In [1]: quickref

This does depend on automagic functionality being enabled, so if you see an error when issuing the above command, you may need to enable automagics by running %automagic on first.

Exploring Code Objects

On Day 2, you learned how to view information about code objects using ? and ??. For example, you can view information about the built-in print() function by typing print? and pressing Enter:

In [2]: print?
Docstring:
print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file:  a file-like object (stream); defaults to the current sys.stdout.
sep:   string inserted between values, default a space.
end:   string appended after the last value, default a newline.
flush: whether to forcibly flush the stream.
Type:      builtin_function_or_method

IPython includes magic commands for viewing specific types of information about code objects. For instance, the %pdoc magic displays just the docstring for an object:

In [3]: %pdoc print
Class docstring:
    print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

    Prints the values to a stream, or to sys.stdout by default.
    Optional keyword arguments:
    file:  a file-like object (stream); defaults to the current sys.stdout.
    sep:   string inserted between values, default a space.
    end:   string appended after the last value, default a newline.
    flush: whether to forcibly flush the stream.
Call docstring:
    Call self as a function.

You can use %pdef to see the call signature of a function or other Python callable object:

In [4]: %pdef enumerate
Class constructor information:
 enumerate(iterable, start=0)

Use %pfile to view the file that an object is defined in:

In [5]: import pathlib
In [6]: %pfile pathlib.Path

The file is opened up at the line where the object is defined and displayed full screen:

You can use PgUp, PgDn, and the arrow keys to navigate the file. You can't edit the file this way, but you can at least view the source, which is handy if you want to see how something works under the hood, or just need to remind yourself of how something is implemented.

If no file exists for the object, then IPython will tell you:

In [7]: %pfile zip
No file found for zip

Out of the three magic commands mentioned here, %pdoc and %pfile are perhaps the most useful. Most of the time, you'll probably just want to use ? to get more information about an object. Other times, it's nice to just view the docstring using %pdoc. And having quick access to the source code for an object with %pfile greatly reduces the number of times you need to switch apps or terminal windows.

Viewing Interactive Variables

During interactive coding sessions, it's common for lots of names to end up in your namespace. All of these names can be difficult to keep track of, and accidentally overwriting one of them could break your entire workflow and require you to spend significant time recreating state.

IPython provides two magic commands that help you keep track of your interactive variables:

  1. %who can print all interactive variables for you, or only certain types of variables
  2. %whos prints interactive variables and includes additional information such as the type of object assigned to that variable and its value.

For example, suppose you have a few variables assigned and have defined a function:

In [1]: name = "David"

In [2]: n1, n2 = 1, 2

In [3]: def greet(name):
   ...:     print(f"Hello, {name}")
   ...:

Running %who shows you all of the variables and names in your namespace:

In [4]: %who
greet	 n1	 n2	 name

You can show only functions by passing the word function to %who:

In [5]: %who function
greet

Using %whos will give you even more details about each variable:

In [6]: %whos
Variable   Type        Data/Info
--------------------------------
greet      function    <function greet at 0x1054b7370>
n1         int         1
n2         int         2
name       str         David

Like %who, you can also search for specific types of variables:

In [7]: %whos int
Variable   Type    Data/Info
----------------------------
n1         int     1
n2         int     2

%who and %whos are handy magics to have built into your muscle memory, so I suggest you spend some time practicing using them.

Configuring IPython With Magic Commands

On Day 2, you learned how to open IPython's configuration file. Some of the configuration settings are available for editing from within the IPython REPL itself. So, if you ever need to change IPython's configuration in the middle of an interactive session, you might be able to do so without opening a text editor.

For instance, you can change the color scheme using the %colors magic command:

In [8]: %colors nocolor    # Change to black and white

In [9]: %colors linux      # Works on dark background

In [10]: %colors lightbg   # Works on light background

Changing the color scheme with %colors works only as long as the current REPL session or until you use %colors to change to a new color scheme. IPython's configuration file does not get changed, so the choice of color scheme does not persist between sessions.

Use the %config magic to expose parts of IPython's configuration system. Running %config without any arguments will show you what configuration classes are available:

In [11]: %config
Available objects for config:
    AliasManager
    DisplayFormatter
    HistoryManager
    IPCompleter
    LoggingMagics
    MagicsManager
    OSMagics
    PrefilterManager
    ScriptMagics
    StoreMagics
    TerminalIPythonApp
    TerminalInteractiveShell

To see all of the available options on a class, you can pass just the class name to %config:

In [12]: %config TerminalInteractiveShell
TerminalInteractiveShell(InteractiveShell) options
------------------------------------------------
TerminalInteractiveShell.ast_node_interactivity=<Enum>
    'all', 'last', 'last_expr' or 'none', 'last_expr_or_assign' specifying which
    nodes should be run interactively (displaying output from expressions).
    Choices: any of ['all', 'last', 'last_expr', 'none', 'last_expr_or_assign']
    Current: 'last_expr'
TerminalInteractiveShell.ast_transformers=<list-item-1>...
    A list of ast.NodeTransformer subclass instances, which will be applied to
    user input before code is run.
    Current: []
TerminalInteractiveShell.auto_match=<Bool>
    Automatically add/delete closing bracket or quote when opening bracket or
    quote is entered/deleted. Brackets: (), [], {} Quotes: '', ""
    Current: False
~ Output shortened ~

The output also shows you what the current values of the configuration settings are. For example, the TerminalInteractiveShell.auto_match setting is current;y set to False. That setting controls whether or not IPython automatically adds closing brackets or quotes when an opening bracket or quote is entered.

To set TerminalInteracticeShell.auto_match to True, pass the entire assignment expression to %config:

In [13]: %config TerminalInteractiveShell.auto_match = True

Typing out the long IPython configuration class names is clunky and error-prone. Fortunately, IPython's tab completion system works here:

Just like %colors doesn't persist color scheme changes between sessions, %config doesn't write any changes to the IPython configuration file. Any changes you make to the configuration will be reset the next time you close and open IPython again.

Using pip Inside of IPython

One of the more useful magic commands is the %pip magic, which allows you to install packages with pip from right inside of the IPython REPL:

In [1]: %pip install requests
Collecting requests
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 1.7 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 2.0 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.0/139.0 KB 4.2 MB/s eta 0:00:00
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Installing collected packages: certifi, urllib3, idna, charset-normalizer, requests
Successfully installed certifi-2021.10.8 charset-normalizer-2.0.12 idna-3.3 requests-2.27.1 urllib3-1.26.9
Note: you may need to restart the kernel to use updated packages.

In some cases, you'll need to restart IPython in order for the freshly installed package to work.

All of the pip commands that you know and love work inside of IPython. For instance, you can run %pip list to see the current packages installed in your Python environment:

In [2]: %pip list
Package            Version
------------------ ---------
appnope            0.1.3
asttokens          2.0.5
backcall           0.2.0
certifi            2021.10.8
charset-normalizer 2.0.12
decorator          5.1.1
executing          0.8.3
idna               3.3
ipython            8.2.0
jedi               0.18.1
matplotlib-inline  0.1.3
parso              0.8.3
pexpect            4.8.0
pickleshare        0.7.5
pip                22.0.4
prompt-toolkit     3.0.29
ptyprocess         0.7.0
pure-eval          0.2.2
Pygments           2.11.2
requests           2.27.1
setuptools         61.3.1
six                1.16.0
stack-data         0.2.0
traitlets          5.1.1
urllib3            1.26.9
wcwidth            0.2.5
wheel              0.37.1
Note: you may need to restart the kernel to use updated packages.

But which environment did requests get installed in? If you exit IPython and run pip list in your shell, you might see a different set of packages:

$ pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 58.1.0

Since we installed IPython using pipx on Day 1, the %pip command uses the isolated environment that pipx created for IPython instead of the global Python environment. This means that any package installed using %pip will be available from within IPython without polluting your global environment.

Of course, this behavior isn't always what you want. You may want to install something in your global environment. Or, perhaps more likely, you'll want to install a package in an active virtual environment.

In those cases, use !pip install to install the package. Rather than using the isolated environment created by pipx, packages will be installed in whatever the currently active Python environment is in your shell.

⚠️
In order to make sure the right pip is used, you may want to use the !python3 -m pip command instead. Brett Cannon's article Why You Should use python -m pip gives a great explanation for this pattern.

Day 4 Activity

Continue to use IPython for all of your daily tasks. Practice using magic commands. Use the configuration magic commands to explore and play around with different configuration options.

Enjoying this course? Why not share it with a friend or colleague that you think will benefit from it? Just right-click on this link, copy the URL, and send it to them in an email!