xonsh

Bringing Python data science to your shell!

Install & Setup

Install

] .right-column[ First, let's install xonsh (if you haven't already).

conda:

$ conda install -c conda-forge xonsh

or, if you must, pip:

$ pip install "xonsh[pygments,ptk,<linux|mac|win>]"

.footnote[.red[*] Also, make sure you are on Python 3] ]

Install & Setup

Install

Setup

] .right-column[ Now to startup xonsh, simply run the xonsh command,

$ xonsh
user@shell ~ $

]

Install & Setup

Install

Setup

Sanity Check

] .right-column[ Now to check that everything is working, run xonfig,

$ xonfig
+------------------|----------------------+
| xonsh            | 0.9.6.dev33          |
| Git SHA          | 626b94db             |
| Commit Date      | Jun 28 17:20:28 2019 |
| Python           | 3.7.3                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 2.0.9                |
| shell type       | prompt_toolkit2      |
| pygments         | 2.4.2                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | ubuntu               |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
+------------------|----------------------+

]

Questions?

Agenda

Xonsh Language Basics
The Environment
Callable Aliases
Events
Macros
Advanced Configuration
Case Study ]

Xonsh Language Basics

Xonsh as a Python Interpreter

--

--

Adding numbers together
Opening Files
Importing modules
Defining functions & classes

]

--

.big[Let's try a few of these out!]

Exercises

At the command line,

Compute the product of 2, 3, & 7.
```
$ 2 * 3 * 7
42
```

Import NumPy (or sys, if you don't have NumPy installed)

$ import numpy as np
$ np.arange(1, 6)
array([1, 2, 3, 4, 5])

Define an add() function, which adds two numbers together and use it to sum together 2842 and 1400.
```
$ def add(x, y):
.     return x + y
.
$ add(2842, 1400)
4242
```

Xonsh as a Shell

--

.big[More exactly, xonsh is a scripting language that is mostly compatible with Unix sh-lang.red[*].]

.footnote[.red[*] If you don't know sh, don't worry! The point of xonsh is to replace the hard-to-learn, hard-to-remember bits of sh with Python.]

--

.big["Why isn't xonsh also a sh-lang superset?"]

--

.big[It is impossible to be both a Python superset and a sh superset, and we have one simple rule:]

--

Running Commands

--

# a simple echo
$ echo "Shello, World!"
Shello, World!

# create a file in a new directory
$ mkdir -p snail
$ cd snail
$ touch deal-with-it.txt
$ ls
deal-with-it.txt

# simple piping example
$ echo "Wow Mom!" | grep o
Wow Mom!

--

Environment Variables

Environment variables are written as a $ followed by a name.

$ $HOME
'/home/it-me'

--

You can set (and export) environment variables with normal Python syntax:

$ $GOAL = "don't panic"
$ print($GOAL)
don't panic
$ del $GOAL

--

Coming from Python, these should look and feel like any other variable, except that they live in the environment.

Environment Lookup with `${expr}`

Sometimes you might not know the name of the environment variable, or you might want to compute the name programatically, or maybe it has special characters in it.

--

In these cases, xonsh provides the ${expr} syntax.

--

This lets you use a Python expression to lookup the value of the environment variable.

--

$ x = "USER"
$ ${x}
'it-me'
$ ${"HO" + "ME"}
'/home/it-me'

--

.red[Warning!] In sh-langs, $NAME and ${NAME} mean the same thing. In xonsh, they have distinct meanings.

The Environment Itself `${...}`

If you ever need access to the environment object, you can grab it by passing in an ellipsis as the lookup expression, i.e. ${...}.

--

$ ${...}
xonsh.environ.Env(
{'AUTO_CD': False,
 'AUTO_PUSHD': False,
 'AUTO_SUGGEST': True,
 'AUTO_SUGGEST_IN_COMPLETIONS': False,
 'BASH_COMPLETIONS': EnvPath(
['/usr/share/bash-completion/bash_completion']
),
 'BASH_COMPLETION_USER_DIR': '/home/scopatz/miniconda/share/bash-completion',
 'BOTTOM_TOOLBAR': '',
 ...
})

--

This is a reference to the object that lives at __xonsh__.env.

--

You can use this object to test if a variable exists:

$ "PATH" in ${...}
True

The `source` command

The source command:

Reads in the contents of a file,
Executes it, and
Brings all global variables into the current execution context.

--

example.xsh:

$EMAIL = "snail@xon.sh"

password = "swordfish"

def combine():
    return $EMAIL + ":" + password

--

Running this:

$ source example.xsh
$ combine()
'snail@xon.sh:swordfish'

As Python, you could also import example, with the normal Python rules.

???

Context is tied correctly

$ password = "monkey"
$ combine()
'snail@xon.sh:monkey'

`source-foreign`

Unlike other languages, xonsh enables you to source scripts written in shells other than xonsh (or Python).

--

The most common of these is Bash.

example.sh

export WAKKA="jawaka"

function now_playing {
  echo "$(whoami) on drums!"
}

--

Running this with source-foreign bash or the source-bash shortcut:

$ source-bash example.sh
$ $WAKKA
'jawaka'
$ now_playing
it-me on drums!

--

Also stock support for source-zsh and source-cmd!

The `xonshrc` configuation file

--

--

--

~/.xonshrc

$MULTILINE_PROMPT = '`·.,¸,.·*¯`·.,¸,.·*¯'
$XONSH_SHOW_TRACEBACK = True

--

.big[A complete listing of environment variables that xonsh knows about is available at https://xon.sh/envvars.html]

Exercises

Set a random integer to the environment variable $SECRET.
```
import random
$SECRET = random.randint(1, 42)
```

Print the secret value after the phrase My secret value is:

# Solution 1 $ echo "My secret value is $SECRET" Solution 2

$ print("My secret value is", $SECRET)

In your xonshrc, generate a secret value. Print this value when xonsh starts up if another $SAFE environment variable does not exist.
~/.xonshrc
```
import random
$SECRET = random.randint(1, 42)
if "SAFE" not in ${...}:
    echo "My secret value is $SECRET"
```

Mixing Python and Subprocess Modes

Subprocess operators

xonsh makes using subprocess simple.

Captured subprocesses

`$()`

] .right-column[

$() captures the standard output of a command and returns it as a string.

]

--

$ x = $(ls)
$ x
'LICENSE\nREADME.md\nascii_conch_part_transparent_tight.png\ndefault.css\nfavicon.ico\nremote.html\ntutorial.md\n'
$ print(x)
LICENSE
README.md
ascii_conch_part_transparent_tight.png
default.css
favicon.ico
remote.html
tutorial.md

]

Captured subprocesses

`$()`

] .right-column[

And it's a str with all the methods you would expect:

]

--

$ x.upper()
'LICENSE\nREADME.MD\nASCII_CONCH_PART_TRANSPARENT_TIGHT.PNG\nDEFAULT.CSS\nFAVICON.ICO\nREMOTE.HTML\nTUTORIAL.MD\n'
$ x.split("\n")
['LICENSE',
 'README.md',
 'ascii_conch_part_transparent_tight.png',
 'default.css',
 'favicon.ico',
 'remote.html',
 'tutorial.md',
 '']

]

Captured subprocesses

`$()`

`!()`

] .right-column[ !() captures command output and a lot of other info, too!

Beyond the output, !() offers the return code, process id, stderr and stdout streams, whether the cmd is an alias, timestamps, and more!

]

--

$ x = !(ls)
$ x
CommandPipeline(stdin=<_io.BytesIO object at 0x7fc0d01be4c0>, stdout=<_io.BytesIO object at 0x7fc0d01be6d0>, stderr=<_io.BytesIO object at 0x7fc0d01bea40>, pid=16969, returncode=0, args=['ls'], alias=['ls', '--color=auto', '-v'], stdin_redirect=['<stdin>', 'r'], stdout_redirect=[10, 'wb'], stderr_redirect=[12, 'w'], timestamps=[1561758574.868479, 1561758575.6660624], executed_cmd=['/usr/bin/ls', '--color=auto', '-v'], input=, output=LICENSE
README.md
ascii_conch_part_transparent_tight.png
default.css
favicon.ico
remote.html
tutorial.md
, errors=None)

]

Captured subprocesses

`$()`

`!()`

] .right-column[

!() returns an instance of a CommandPipeline object -- this object is "truthy" if the command completed successfully:

]

--

$ bool(!(ls .))
True
$ bool(!(ls nothingtoseehere))
/usr/bin/ls: cannot access 'nothingtoseehere': No such file or directory
False

]

Captured subprocesses

`$()`

`!()`

] .right-column[ You can iterate over the output from !() line-by-line:

$ for i, loc in enumerate(!(ls)):
°     print(f"{i}th: {loc.strip()}")
°
°
0th: LICENSE
1th: README.md
2th: ascii_conch_part_transparent_tight.png
3th: default.css
4th: favicon.ico
5th: remote.html
6th: tutorial.md

]

Uncaptured subprocesses

`$()`

`!()`

`$[]`

] .right-column[

$[] and ![] are the uncaptured subprocess expressions. They are similar to $() and !() with the notable difference that they stream output to stdout.

]

Uncaptured subprocesses

`$()`

`!()`

`$[]`

] .right-column[ $[] always returns None.

You may be asking, what is this even for?

More on that later, but the short answer is that it allows you to force xonsh to recognize a command as a subprocess command if the context is ambiguous.

$ x = $[ls .]
LICENSE    ascii_conch_part_transparent_tight.png  favicon.ico  tutorial.md
README.md  default.css

]

Uncaptured subprocesses

`$()`

`!()`

`$[]`

`![]`

] .right-column-tight[ ![] streams command output to stdout but also returns an instance of a HiddenCommandPipeline object.

$ x = ![ls .]
LICENSE    ascii_conch_part_transparent_tight.png  favicon.ico  tutorial.md
README.md  default.css                             remote.html

]

--

$ x.args
['ls', '.']
$ x.timestamps
[1561759961.047233, 1561759961.0570006]
$ x.alias
['ls', '--color=auto', '-v']

]

Uncaptured subprocesses exercises

`$()`

`!()`

`$[]`

`![]`

] .right-column-tight[

Take a few minutes to try out each of the subprocess operators.

Use xonsh's tab completion to explore the attributes of the CommandPipelines and get a sense of what information is in there.

]

Subprocess Operators: A helpful mnemonic(?)

`$()`

`!()`

`$[]`

`![]`

] .right-column[

"Curly Captures"

]

--

"Square Streams"

]

The Python-mode operator

`@()`

]

The @() operator allows us to insert Python variables and values into subprocess commands.

`xonsh` can always mix and match subprocess and Python commands without additional syntax if those commands are on separate lines, e.g. ]

$ for _ in range(2):
°     echo "Hi!"
°
Hi!
Hi!

]

The Python-mode operator

`@()`

] .right-column[

But what about passing a variable to `echo` (or any other subprocess command?) ]

$ for i in range(2):
°     echo i
°
i
i

]

The Python-mode operator

`@()`

] .right-column[

The `@()` operator evaluates arbitrary Python expressions and returns the result as string. This result can be fed directly to a subprocess command: ]

$ for i in range(2):
°     echo @(i)
°
0
1

]

If the output is a non-string sequence (list, set, etc.) then the results are joined and returned as a string.

$ echo @([x for x in range(3)])
0 1 2

]

The Python-mode operator

`@()`

] .right-column[

If the result of `@()` is in the first position it is treated as an alias: ]

$ @("echo hello there".split())
hello there

]

(strings are not split automatically)

$ @("echo hello there")
xonsh: subprocess mode: command not found: echo hello there

]

Regex ticks

``

]

Ready for super-charged file matching? ]

You can wrap a regular expression in `` and it will return a list of matching files and directories. ]

$ `.*.md*`
['README.md', 'tutorial.md']

]

The ticks are also a Python expression so you can use them in for loops, or list-comprehensions, or whatever floats your boat.

$ [f.lower() for f in `.*.md`]
['readme.md', 'tutorial.md']

]

Glob ticks

``

g``

]

If you prefer globs over regex, prepend a g to your tick expression:

$ g`*.md`
['README.md', 'tutorial.md']

]

Glob ticks also support recursive globbing with double **:

$ g`**/*.md`

]

`xonsh` string literals

`f""`

] .right-column[

f-strings, or formatted string literals, are a part of Python 3.6+ and they are great. ]

$ x = 5
$ print(f"x is {x}")
x is 5

]

xonsh supports f-strings (so long as your underlying Python is 3.6+) and it also has a few extra tricks up its sleeves!

]

path string literals

`f""`

`p""`

] .right-column[

p-strings are a `xonsh` feature that allow easy construction of `pathlib.Path`s. Any string that has a leading `p` becomes a `Path`. ]

$ path = p"my_cool_folder"
$ path
PosixPath('my_cool_folder')
$ path.exists()
False

]

If you haven't used pathlib before, take a moment to look through all of the Path attributes and methods -- they're super useful!

Also try out using the / operator with a pathlib.Path.

Note: Your OS will determine what sort of path object you get.

]

formatted path string literals

`f""`

`p""`

`pf""`

] .right-column[

There are p-strings and f-strings, which leads to a natural question -- what about pf-strings? ]

$ home = "home"
$ user = "gil"
$ gitdir = "github.com"
$ p = pf"/{home}/{user}/{gitdir}"
$ p
PosixPath('/home/gil/github.com')

]

But wait! Environment variables are also Python objects: ]

$ p = pf"{$HOME}/{gitdir}"
$ p
PosixPath('/home/gil/github.com')

Cool, huh? ]

Exercises

Questions

The Environment

Environment

All of xonsh's environment variables live in __xonsh__.env.

You can also access the environment using ${...}.

--

You can use the membership operator to determine if an environment variable is present in your current session.

$ 'HOME' in ${...}
True

--

If you want more information about a certain environment variable in xonsh, you can ask for help!

$ ${...}.help("AUTO_CD")
AUTO_CD:

Flag to enable changing to a directory by entering the dirname or full path
only (without the cd command).

default: False
configurable: True

Typing & Detyping

xonsh environment variables are Python objects. This means they also have types like Python objects. Sometimes these types are imposed based on a variable name.

--

Any env-var matching \w*PATH will be of type EnvPath, like

PATH
LD_LIBRARY_PATH
RPATH

--

Other variables are boolean, others are ints. Whatever their type, in xonsh you always have access to the true object, not a string representation.

Detyping

xonsh automatically converts variables to strings whenever it feeds them to subprocess commands. You can also explicitly request detyped variables:

--

$ ${...}.get("PATH")
EnvPath(
['/opt/miniconda3/bin/',
 '/usr/bin',
 '/usr/local/bin',
 '/usr/bin/vendor_perl/',
 '/usr/bin/core_perl/']
)

--

$ ${...}.detype().get("PATH")
'/opt/miniconda3/bin/:/usr/bin:/usr/local/bin:/usr/bin/vendor_perl/:/usr/bin/core_perl/'

Environment Swapping

There are a couple of other useful methods on ${...}, in particular, the swap() method is useful for temporarily setting environment variables.

--

$ with ${...}.swap(SOMEVAR='foo'):
°     echo $SOMEVAR
°
foo
$ echo $SOMEVAR
$SOMEVAR

Exercises

Try using getpass.getpass() with swap() to set your "password" in an environment variable temporarily.


$ import getpass
$ with ${...}.swap(PASS=getpass.getpass()):
°     echo @(f"git push https://gil:{$PASS}@github.com/xonsh/xonsh master")
°
Password:
git push https://gil:hunter2@github.com/xonsh/xonsh master
$ echo $PASS
$PASS

A curl binary in your (conda|homebrew) install is messing with your built-in package manager. It would be handy if you could remove the first entry of your $PATH but only to run the one install command...

$ echo $PATH /opt/miniconda3/bin/:/usr/bin:/usr/local/bin:/usr/bin/vendor_perl/:/usr/bin/core_perl/ $ with ${...}.swap(PATH=$PATH[1:]): ° echo "sudo pacman -S pinentry" ° echo $PATH ° sudo pacman -S pinentry /usr/bin:/usr/local/bin:/usr/bin/vendor_perl/:/usr/bin/core_perl/

$ echo $PATH /opt/miniconda3/bin/:/usr/bin:/usr/local/bin:/usr/bin/vendor_perl/:/usr/bin/core_perl/

Callable Aliases

--

--

.big[Let's fully integrate Python into a data-processing pipeline by allowing functions (callables) to be executed in subprocess mode...]

--

Callable Aliases

.big[This works for any Python function that adheres to a certain set of known signatures. (So not any function.)]

--

.big[These are known as callable aliases because they are often placed in the builtin aliases dictionary, along side the more normal aliases.]

--

.big[By placing them in aliases, they are executable from anywhere, as with normal subproess commands and aliases.]

Function signatures: Nothing!

In the simplest case, the alias takes no arguments and returns a string or an integer return code.

--

$ aliases['banana'] = lambda: "Banana for scale.\n"
$ banana
Banana for scale.

--

We can pipe this to a normal command:

$ banana | wc -w
3

--

And if we want to destoy the alias:

$ del aliases['banana']

Function signatures: The Loneliest

Callable aliases may also optionally take command-line options as the first positional argument.

--

These come in as a list of strings, like sys.argv.

--

def apple(args):
    if len(args) == 1:
        print("all alone with " + args[0])
        return 0
    else:
        print("too much company!\n- " + "\n- ".join(args))
        return 1

--

The callable alias needs to be the first element of a command:

$ @(apple) core
all alone with core

$ @(apple) core seed
too much company!
- core
- seed

Function signatures: Fourtified

Standard streams are also able to be passed in as keyword arguments whose defaults are None.

--

These are file open objects with all of the usual Python interfaces.

--

def _grape(args, stdin=None, stdout=None, stderr=None):
    for line in stdin:
        stdout.write("Grape says '" + line.strip().lower() + "'\n")
    return 0

aliases["grape"] = _grape

--

usage:

$ echo WrATh | grape
Grape says 'wrath'

--

Note that not all streams must be passes in, but the prior streams must be passed in if the latter ones are. For example, if you want to use stdout you have to accept stdin but not stderr.

Function signatures: A Kiwious Spectacle

The command specification spec is available as the next keyword argument.

--

This is a rich Python object that represents the metadata about the callable alias that is running.

--

For example, if the alias wants to add a newline if it is uncaptured, but ignore the newline otherwise, you need the captured attr of the spec.

def _kiwi(args, stdin=None, stdout=None, stderr=None, spec=None):
    import xonsh.proc
    if spec.captured in xonsh.proc.STDOUT_CAPTURE_KINDS:
        end = ''
    else:
        end = ' (newline)\n'
    print('Hi Mom!', end=end)
aliases["kiwi"] = _kiwi

--

$ kiwi  # uncaptured
Hi Mom! (newline)


$ $(kiwi)  # captured
'Hi Mom!'

Function signatures: Framed!

The final form takes a stack argument as well.

--

This is a list of FrameInfo instances, and is for the truly maniacal.

--

The stack argument provides all the alias with everything it could possible want to know about its call site.

--

def lemon(args, stdin=None, stdout=None, stderr=None, spec=None, stack=None):
    for frame_info in stack:
        frame = frame_info[0]
        print('In function ' + frame_info[3])
        print('  locals', frame.f_locals)
        print('  globals', frame.f_globals)
        print('\n')
    return 0

--

Exercises

Write a callable alias frankenstein, which provides the content of Mary Shelley's classic novel, full text here.

# short version
aliases["frankenstein"] = lambda: $(curl https://www.gutenberg.org/files/84/84-0.txt)
streaming version

def _frankenstein(args, stdin=None, stdout=None):
for line in !(curl https://www.gutenberg.org/files/84/84-0.txt):
stdout.write(line)
aliases["frankenstein"] = _frankenstein

Write a callable alias upper, that uppercases what it reads on stdin.

# short version
aliases["upper"] = lambda args, stdin: stdin.read().upper()
streaming version

def _upper(args, stdin=None, stdout=None):
for line in stdin:
stdout.write(line.upper())
aliases["upper"] = _upper

Write a callable alias words, that returns all of the unique, sorted words coming from stdin.
```
aliases["words"] = lambda args, stdin: "\n".join(sorted(set(stdin.read().split())))
```
Write a callable alias count, that returns the number of tokens read from stdin.
```
aliases["count"] = lambda args, stdin: str(len(stdin.read().split())) + "\n"
```
Combine the above aliases in a single pipeline to count the number of words in "Frankenstein."
```
$ frankenstein | upper | words | count
11724
```

Break

Events

Event Handlers

Yay for events! An event is an... event?

An event is a trigger that can fire at predefined points in code.

xonsh has a simple, but powerful, event system that you can use to peek inside of existing programs or to add more functionality to your own.

--

An event handler is a function that is called whenever a given event is fired. An event can have multiple handlers -- that is, you can run arbitrarily many functions that are all triggered by a single event firing.

Let's look at an example!

`events.on_chdir`

There are several events that are already registered with xonsh (we'll cover how to add new events in a little bit).

events.on_chdir fires every time we... change directory. Because this event already exists, all we want to do is to register a handler to listen for the event and then execute.

--

$ @events.on_chdir
° def chdir_handler(olddir, newdir):
°     print(f"The directory changed from {olddir} to {newdir}!")
°

--

$ cd ..
The directory changed from /home/gil/github.com/xonsh/scipy-2019-tutorial to /home/gil/github.com/xonsh!
$ cd scipy-2019-tutorial/
The directory changed from /home/gil/github.com/xonsh to /home/gil/github.com/xonsh/scipy-2019-tutorial!

Event Management

Um... how do I make this stop?

That was a verbose event handler we created. Let's turn it off.

Each event has an associated set of handlers. The simplest way to remove a handler is to pop it off.

$ events.on_chdir.pop()
<function __main__.chdir_handler>

Now that we can move around without the shell yelling at us, let's look at that example in-depth.

Handler Registration

You can register a handler by decorating a function with the event object:

$ @events.on_chdir
° def chdir_handler(olddir, newdir):
°     print(f"The directory changed from {olddir} to {newdir}!")

Note that olddir and newdir weren't specified by us - they're supplied by the event itself.

--

Not all events provide arguments to their handlers -- you can check by calling help(event.name), although that's overly verbose.

For now, it's simpler to print the __doc__ directly:

$ print(events.on_chdir.__doc__)

on_chdir(olddir: str, newdir: str) -> None

Fires when the current directory is changed for any reason.

The type-hint-esque signature tells us which variable names to expect and their types. (If you don't want to deal with them, you can capture them using **kwargs)

When events are fired

Some events are fired because xonsh has them set up to fire already. Others fire when you tell them to. Many event names are self-descriptive, but if there's any ambiguity, you can always check the __doc__.

You can also look at the list of events in the docs

Let's walk through defining our own event, setting it to fire, then set up a handler to react to it.

Example Event

First we have to create the new event. Believe it or not, to create an event, you create a docstring for the event. This is truly self-documenting code.

Let's create an event that raises an alarm if it's called. This is our first event, so we won't add in any kwargs:

--

$ events.doc("never_run_this", """
° never_run_this() -> None
°
° Fired when forbidden functions are run.
° """)

--

Now write a function that fires the event. This function could do a bunch of other stuff, but for this example we'll keep it simple:

--

$ def delete_my_computer():
°     events.never_run_this.fire()
°

Now run delete_my_computer. What happened?

--

Nothing.

Example Event continued

Not nothing, truly. The event did fire, but we weren't listening. What do we need to add?

--

A handler!

--

Something suitably chastening -- this person did just delete your computer, after all.

--

$ @events.never_run_this
° def WHO_DID_IT():
°     print(f"omg user {$(whoami)} DELETED YOUR COMPUTER")

--

Now, go ahead and delete your computer again:

--

$ delete_my_computer()
omg user gil
 DELETED YOUR COMPUTER

Event exercises

"Fix" the previous on_envvar_change example to only print new environment variables if they aren't CWD or OLDCWD


@events.on_envvar_change
def print_env(name, oldvalue, newvalue):
    if name not in ["CWD", "OLDCWD"]:
        print(f"envvar {name} changed from {oldvalue} -> {newvalue}")

Look at the __doc__ for events.on_postcommand and use it to create a post command hook that sets the $RIGHT_PROMPT to display the starting and ending timestamps of the previous command. ($RIGHT_PROMPT needs to be a string)
```
@events.on_postcommand
def update_rprompt(ts, **kwargs):
    $RIGHT_PROMPT = f"{ts[0]} -> {ts[1]}"
```

--

pop the on_envvar_change handler off and update it to also ignore RIGHT_PROMPT


events.on_envvar_change.pop()
@events.on_envvar_change
def print_env(name, oldvalue, newvalue):
if name not in ["CWD", "OLDCWD", "RIGHT_PROMPT"]:
print(f"envvar {name} changed from {oldvalue} -> {newvalue}")

Macros

What is a Macro

.big[A macro is syntax that replaces a smaller amount of code with a larger expression, syntax tree, code object, etc. after the macro has been evaluated.]

.big[The macro algorithm:]

Pause or skip normal parsing ]

Pause or skip normal parsing
Gather macro inputs as strings ]

Pause or skip normal parsing
Gather macro inputs as strings
Evaluate macro with inputs ]

Pause or skip normal parsing
Gather macro inputs as strings
Evaluate macro with inputs
Resume normal parsing and execution. ] --

.big[Xonsh's macro system is more like Rust's than those in C, C++, Fortran, Lisp, Forth, Julia, etc.]

--

What is a Macro

--

--

.bigger.center[Subprocess Macros]

--

.bigger.center[Function Macros]

--

.bigger.center[Context Macros]

Subprocess Macros

Subprocess macros turn everything after an ! in a subprocess call into a single string argument that is passed into the command.

--

A simple example:

$ echo! I'm Mr. Meeseeks.
I'm Mr. Meeseeks.

--

Xonsh will split on whitespace, so each argument is passed in separately.

$ echo x  y       z
x y z

--

Space can be preserved with quotes, but that can be annoying.

$ echo "x  y       z"
x  y       z

--

Subprocess macros will pause and then strip all input after !.

$ echo! x  y       z
x  y       z

Subprocess Macros

Macros pause all syntax, even fancy xonsh bits like environment variables.

--

For example, without macros, environment variable are expanded:

$ echo $USER
it-me

Inside of a macro, all parsing is turned off.

$ echo! $USER
$USER

--

You can put the ! anywhere in the subprocess:

$ echo I am $USER ! and I still live at $HOME
I am it-me and I still live at $HOME

Subprocess Macros

This is particularly useful whenever you need to write code as string and hand that source off to a command.

--

timeit:

$ timeit! "hello mom " + "and dad"
100000000 loops, best of 3: 8.24 ns per loop

--

bash (for shame!):

$ bash -c ! export var=42; echo $var
42

--

python (yay 🎉):

$ python -c ! import os; print(os.path.abspath("/"))
/

Function Macros

Xonsh supports Rust-like macros that are based on normal Python callables.

--

Macros do not require a special definition in xonsh.

--

However, they must be called with ! between the callable and the opening parentheses (.

--

Macro arguments are split on the top-level commas ,.

--

For example functions f and g:

# No macro args
f!()

# Single arg
f!(x)
g!([y, 43, 44])

# Two args
f!(x, x + 42)
g!([y, 43, 44], f!(z))

Function Macros

--

.big[Arguments in the macro call are matched up with the corresponding parameter annotation in the callable’s signature.]

--

def identity(x : str):
    return x

--

.big[Calling this normally in Python will return the same object we put in, even if it is not a string!]

Function Macros

>>> identity('me')
'me'

>>> identity(42)
42

>>> identity(identity)
<function __main__.identity>

>>> identity!('me')
"'me'"

>>> identity!(42)
'42'

>>> identity!(identity)
'identity'

Function Macros

--

--

>>> identity!(42)
'42'

>>> identity!(  42 )
'42'

--

Function Macros

>>> identity!(import os)
'import os'

>>> identity!(if True:
...     pass)
'if True:\n    pass'

>>> identity!(std::vector<std::string> x = {"yoo", "hoo"})
'std::vector<std::string> x = {"yoo", "hoo"}'

Function Macros Annotations

Category	Object	Flags	Modes	Returns
String	`str`	`'s'`, `'str'`, or `'string'`		Source code of argument as string, default.
AST	`ast.AST`	`'a'` or `'ast'`	`'eval'` (default), `'exec'`, or `'single'`	Abstract syntax tree of argument.
Code	`types.CodeType` or `compile`	`'c'`, `'code'`, or `'compile'`	`'eval'` (default), `'exec'`, or `'single'`	Compiled code object of argument.

--- # Function Macros Annotations

Category	Object	Flags	Modes	Returns
Eval	`eval` or `None`	`'v'` or `'eval'`		Evaluation of the argument.
Exec	`exec`	`'x'` or `'exec'`	`'exec'` (default) or `'single'`	Execs the argument and returns None.
Type	`type`	`'t'` or `'type'`		The type of the argument after it has been evaluated.

--- # Function Macros Annotations

def func(a, b : 'AST', c : compile):
    pass

--

a will be evaluated with str since no annotation was provided,
b will be parsed into a syntax tree node, and
c will be compiled into code object since the builtin compile() function was used as the annotation. ]

--

.center.large[✨🤯✨]

Context Macros

Context macros use with! to capture everything after the colon in an otherwise normal with-statement.

--

This provides anonymous & onymous blocks.

--

with! x:
    y = 10
    print(y)

--

This can be thought of as equivalent to:

x.macro_block = 'y = 10\nprint(y)\n'
x.macro_globals = globals()
x.macro_locals = locals()
with! x:
    pass

--

The macro_block string is dedented.

--

The macro_* attrs are set before the context manager is entered so the __enter__() method may use them.

--

The macro_* attributes are not cleaned up likewise for __exit__() method.

Context Macros

--

.big[However, like with function macro arguments, the kind of macro_block is determined by a special annotation.]

--

--

Context Macros Example

import xml.etree.ElementTree as ET

class XmlBlock:

    # make sure the macro_block comes back as a string
    __xonsh_block__ = str

    def __enter__(self):
        # parse and return the block on entry
        root = ET.fromstring(self.macro_block)
        return root

    def __exit__(self, *exc):
        # no reason to keep these attributes around.
        del self.macro_block, self.macro_globals, self.macro_locals

--

with! XmlBlock() as tree:
    <note>
      <to>You</to>
      <from>Xonsh</from>
      <heading>Don't You Want Me, Baby</heading>
      <body>
        You know I don't believe you when you say that you don't need me.
      </body>
    </note>

print(tree.tag)  # will display 'note'

Exercises

Run the timeit command on formatting the string "the answer is: {}" with the integer value 42 using a subprocess macro.
```
$ timeit! "the answer is: {}".format(42)
10000000 loops, best of 3: 169 ns per loop
```
Call the standard library importlib.import_module() function as a macro to import a module, such as os or sys without an explicit string.
```
$ importlib.import_module!(os)
<module 'os' from '/home/scopatz/miniconda/lib/python3.7/os.py'>
```

Write a JsonBlock contenxt manager that can be used to embed JSON into your xonsh code.

import json
class JsonBlock:
def enter(self):
return json.loads(self.macro_block)
def exit(self, *exc):
del self.macro_block, self.macro_globals, self.macro_locals

Advanced Configuration

xontribs

Tips & Tricks

xontribs

lazy imports

Examples

Case Study

Disentangling results

MRI Data

You are working in a lab that is concerned with MRIs of mouse lemurs.

What's a mouse lemur, you ask?

--

FOCUS, THIS IS SCIENCE

Where were we?

You are working in a lab examining MRI data of mouse lemurs. The data set you need is on a webserver set up by the post-doc who disappeared last week.

--

--

The state of the nation data

The post-doc uses bash to handle all of the data collection.

He didn't know how to make sure different datasets were saved to separate directories, so he added a random 4-digit integer to the end of each nii.gz file.

But, in the script, he neglected to add the same random number to the end of the corresponding json metadata file and... well... we don't know if these are our mouse lemur scans, or if they're cervical spine scans...

--

He also ran rm with an overly permissive glob and deleted all of the json metadata files.

--

To make matters worse -- these files are kind of big, and the post-doc's webserver is at his house. We don't want to have to download all of these files if we don't have to.

But don't despair!

We know a couple of things:

--

All of the files follow the naming convention sub-{:02d}_{:04d}.nii.gz

--

While nii.gz files are a bit large, they have some identifying information in the first 348 bytes of the file

--

We're sure there are 19 mouse lemur scans

--

The cervical spine study going on at the lab next door (hence the overlapping files) only has 6 subjects

--

Plan of action

There is a great Python library for handling MRI data called nibabel -- you can install it by running

conda install -c conda-forge nibabel

Install nibabel
Use curl to get the file-list from the webserver -- it isn't formatted super well and will take some massaging.
Use curl to download only the header-portion of the files
Load the files into nibabel and inspect the header attribute and see if we can figure out how to separate the two sets of scans.

curl -r 0-347 -O path/to/file

will download only the first 348 bytes of a given file and save it with the same name locally.]

Files

tutorial.md

Latest commit

History

tutorial.md

File metadata and controls

xonsh

Install & Setup

Install

.footnote[.red[*] Also, make sure you are on Python 3] ]

Install & Setup

Install

Setup

]

Install & Setup

Install

Setup

Sanity Check

]

Questions?

Agenda

Xonsh Language Basics

Xonsh as a Python Interpreter

.big[Let's try a few of these out!]

Exercises

Xonsh as a Shell

Running Commands

Environment Variables

Environment Lookup with ${expr}

The Environment Itself ${...}

The source command

source-foreign

The xonshrc configuation file

Exercises

Solution 2

Mixing Python and Subprocess Modes

Subprocess operators

Captured subprocesses

$()

Captured subprocesses

$()

Captured subprocesses

$()

!()

Captured subprocesses

$()

!()

Captured subprocesses

$()

!()

Uncaptured subprocesses

$()

!()

$[]

Uncaptured subprocesses

$()

!()

$[]

Uncaptured subprocesses

$()

!()

$[]

![]

Uncaptured subprocesses exercises

$()

!()

$[]

![]

Subprocess Operators: A helpful mnemonic(?)

$()

!()

$[]

![]

"Curly Captures"

"Square Streams"

]

The Python-mode operator

@()

]

xonsh can always mix and match subprocess and Python commands without additional syntax if those commands are on separate lines, e.g. ]

Environment Lookup with `${expr}`

The Environment Itself `${...}`

The `source` command

`source-foreign`

The `xonshrc` configuation file

`$()`

`$()`

`$()`

`!()`

`$()`

`!()`

`$()`

`!()`

`$()`

`!()`

`$[]`

`$()`

`!()`

`$[]`

`$()`

`!()`

`$[]`

`![]`

`$()`

`!()`

`$[]`

`![]`

`$()`

`!()`

`$[]`

`![]`

`@()`

`xonsh` can always mix and match subprocess and Python commands without additional syntax if those commands are on separate lines, e.g. ]

`@()`

But what about passing a variable to `echo` (or any other subprocess command?) ]

`@()`

The `@()` operator evaluates arbitrary Python expressions and returns the result as string. This result can be fed directly to a subprocess command: ]

`@()`

If the result of `@()` is in the first position it is treated as an alias: ]

`xonsh` string literals

`f""`

`f""`

`p""`

p-strings are a `xonsh` feature that allow easy construction of `pathlib.Path`s. Any string that has a leading `p` becomes a `Path`. ]

`f""`

`p""`

`pf""`