Sam's world

I’m Back!

2025-01-23T00:00:00-05:00

I haven’t updated this blog since 2020, but lots of things happened between the end of 2020 and January 2025. I moved across the country from New York and settled down in Vancouver, Washington, at the start of the pandemic. During my time in the Pacific Northwest, besides continuing to expand KIP within Kepler, I made a bunch of friends, did a lot of hiking, played lots of trivia with an ever-rotating group, survived the pandemic, got engaged, and got pretty good at playing ukulele (shout-out to the Doomsday Brewing safehouse, which has since become the Pour Choices Pub, where I spent every Tuesday jamming with a great group).

Near the beginning of 2023, my fiancée and I began discussing whether the Pacific Northwest was for us. Over the next two years, we explored more of Portland and Seattle and ultimately concluded that it was time for us to get closer to immediate family. 2024 was a blur: a cross-country move, selling the house, bidding goodbye to friends, and making new ones.

Which brings me to this blog. Honestly, it’s been so long since I’ve thought about it that I was curious whether it was still functional. It’s a static site hosted on GitHub Pages (source code here), so I figured it probably works. Now that we’re in a more stable place and not constantly exploring, if I’m able to publish this piece, I plan to publish more pieces in line with my interests. Through writing my thoughts down in a regular cadence, I hope I can both connect with folks who are interested in my thoughts and identify exciting new areas of exploration through long-form writing.

But that’s all for now! Gotta go play some pickleball.

How to write a coc.nvim extension

2020-08-14T00:00:00-05:00

Table of Contents

Background
- Language Server Protocol
- VSCode: client power
coc.nvim
Wrapping up

In this post, I build a “coc.nvim” extension that wraps an executable language server. By the end of this article, you should both understand what makes coc a stand-out LSP client and be able to write your own coc extension.

For the interactive parts of this post, you’ll need the following:

A POSIX-compliant terminal (bash, zsh, etc), or the ability to translate
git
Node.js>=8.10.0
yarn>=1.22.4
Vim 8+ or Neovim 0.4.4+
coc.nvim==0.0.78 (might work on newer versions, but no promises)
Some knowledge of TypeScript might be helpful

Please disable Python-specific coc extensions (coc-jedi, etc). Some terminology used throughout the post:

Vim: Vim or Neovim
vimrc: ~/.config/nvim/init.vim for Neovim or ~/.vimrc for Vim
coc-settings.json: ~/.config/nvim/.coc-settings.json by default

Background¶

coc.nvim, short for “conquer of completion”, is an lsp client that targets Vim. Vim (or NeoVim) is my favorite text editor / IDE. I like its extensibility, flexibility, and in-terminal slickness. Vim has tools and plugins that make it easy to program in different programming languages. Here are some examples:

JavaScript intellisense: tern_for_vim
Ruby on rails intellisense: vim-rails
General linting: syntastic
Python intellisense: jedi-vim

Vim’s diverse plugins and community-supported tooling are what keep me coming back, but not all developers are like me. Visual Studio Code, Nodepad++, and others were more popular than Vim in the 2019 StackOverflow developer survey:

These popularity differences are starker across programming language communities. For example, Vim’s Java development environment was historically abysmal enough to generate some heated conversations. My non-scientific evaluation of the community’s sentiment: obstinate, impractical developers used Vim for Java development while practical, no-nonsense creators preferred Eclipse. Because of this sentiment, tooling for Java evolved in the Eclipse world while Vim’s Java support remained neglected.

The Java pattern mentioned in the previous paragraph happens, to some extent, in all programming languages. Developers became drawn to editors best-supported their language of choice: PyCharm/Vim for Python, Eclipse for Java, VSCode for TypeScipt/JavaScript, etc. This, in turn, forced developers into mind-bending context switches when changing languages. To become proficient with a new programming language, a developer needed to learn a whole new text editor and tooling ecosystem! Like asking a Jedi to use another Jedi’s light saber, forcing a developer to learn and use Emacs because it has better support for <insert language here> can evoke uncomfortable feelings (at best):

The rise of full stack development and our general expectation that developers be comfortable writing code in more than one language made it so that by mid 2015, we seemed all-but doomed to a life of eternal context switching.

When society was on the brink, the developers behind VSCode had a big idea: if we separate the tools that understand languages from the tools that edit text, and then give these separate tools a communication protocol, this could enable Vim users to use the same language tools that power Eclipse and VSCode (and vise versa). They called this communication mechanism the “language server protocol”.

Language Server Protocol¶

The “Language Server Protocol” (LSP) is a specification that describes communication between a “language client” and a “language server”. The following definitions come from the lsp spec overview:

Language server: a program that contains language-specific “smarts” that can communicate with development tooling through the LSP
Language client: a program, developed in the environment of a “development tool”, that can send, receive, and act on LSP messages with a “language server”
Development tool: Vim, Neovim, VS Code, Eclipse, etc. Basically, a text editor or IDE that software developers use to develop software

Above, the language client is sitting in the development tool and sending JSON-RPC requests to a language server. All this basically means that your text editor is asking questions to an entity that knows way more about your programming language and responding to its suggested actions. These actions may range from “you should autocomplete some text” to “find this symbol’s definition by opening this file and navigating to this precise position. In short, the orchestrated communication outlined above realizes the following dream:

Language tooling developers write 1 program (the language server) to target every development tool (Vim, VS Code, etc)
Development tool developers write 1 program (the language client) that knows how to communicate with all language servers
Software developers have first-class language support for all languages in their favorite editor; no more text-editor context switching!

The above dream seemed like it would herald a new dawn for developers: we can use our favorite tools and have similar, powerful programming language support! As the auguries looked good and a golden age seemed imminent, VSCode’s popularity began generating some bad, familiar omens…

VSCode: client power¶

The LSP dream described in the previous section was similar to Tim Berners-Lee‘s dream for a world wide web: web clients, supporting web protocols, could enable communication between web servers and a user environment (the operating system). This dream sparked development of all sorts of web clients (browsers) and culminated in the first browser war. According to Wikipedia, during this time, Netscape Navigator and Internet Explorer implemented proprietary html tags like <blink> for Navigator and <marquee> for Internet Explorer. They also rapidly developed their own features and began diverging from W3C‘s recommended spec. This resulted in efficient Internet Explorer HTML being less efficient or broken on Navigator (and vise versa). This forced users to pick browsers based on custom compatibility with their favorite sites and ultimately resulted in United States v. Microsoft Corp. Basically, things got ugly.

Although what is happening today in the text editing community is tamer, it resembles the first browser war in that VSCode has become the de-facto standard for Language Clients. Authors of Language Servers may target VSCode without fully supporting the current LSP spec or by inadvertently supporting VSCode features that are not in the spec. This means that Atom users might be using a correctly-implemented language client that does not work with certain language servers. Additionally, there may be no way to easily configure certain language servers without VSCode’s configuration options. This means that if you want to get the best language support possible across all language servers, as of August 2020, you pretty much need to use something that resembles VSCode.

As the VSCode ecosystem seemed like it had formed an exclusive clique, Qiming zhao (@chemzqm on social media) realized that Vim could join the party by implementing a VSCode bridge for Neovim. Qiming zhao named this LSP bridge “coc.nvim”.

coc.nvim¶

coc.nvim is a language client for Vim that can be configured similarly to VSCode. Vim is configured with Vim script, C, lua (if using Neovim), and any other language if you’re feeling fancy enough these days. VSCode is configured with json and TypeScript. “coc.nvim” makes it so we can configure “coc.nvim”-managed features using json and typescript while still being able to use Vim’s configuration for everything else. Basically, it’s this:

Register a language server¶

Without an extension, coc behaves like a simple language client. For simplicity’s sake, let’s assume we’re all Python developers and that we’d like to use a language server named jedi-language-server with Vim.

Within your current Python environment, install jedi-language-server:

pip install jedi-language-server

Now run jedi-language-server:

jedi-language-server

Notice that the process hangs; it’s waiting to receive standard input and will respond with standard output. This server can run inside of Vim by placing the following code in the languageserver section of your coc-settings.json:

{
  "languageserver": {
    "python": {
      "command": "jedi-language-server",
      "filetypes": ["python"]
    }
  }
}

If you’ve configured coc.nvim and installed jedi-language-server in an accessible location, Vim and jedi-language-server will communicate to provide you autocompletion, goto definition, etc. Hooray!

Manual configuration¶

It turns out there are some practical problems with expecting users to configure jedi-language-server manually:

Remembering to manually install a language server can annoy people; shouldn’t the editor remember to do that?
We’d like to configure the language server; can’t that configuration have pretty autocompletion and look like the configuration I can use with all the other VSCode-like configuration options?
It’s good UX to prevent preventable problems.

This is where coc extensions come into play: they avoid the above problems by providing default configurations and features. Users can seamlessly use a language server with an extension.

Writing an extension¶

We will address 2 requirements in this post:

Prevent the user from needing to write manual configuration for jedi-language-server in coc-settings.json
Allow the user to configure the server as “disabled” if needed

The reader may address other requirements as a post-blog exercise.

Project skaffolding¶

In ~/src, use create-coc-extension to create a project skeleton:

$ yarn create coc-extension coc-jls
...
? Project title: coc-jls
? Project description: A great project
? Author full name: Your name
? Author email address: your.name@domain.com
? Initialize a git repository? Yes
? Install node dependencies? Yes
  installing node dependencies...
Done...
$ cd coc-jls
$ yarn

You should have the following version-controlled files in coc-jls:

LICENSE
README.md
package.json
src/
  index.ts
  lists.ts
tsconfig.json
webpack.config.js
yarn.lock

Delete the files in src; we’ll be writing these from scratch.

First file¶

To get a simple wrapper around an existing jedi-language-server executable, place the following code in src/index.ts:

import { ExtensionContext, services, workspace, LanguageClient } from 'coc.nvim'

export async function activate(context: ExtensionContext): Promise<void> {
  const serverOptions = {
    command: 'jedi-language-server', // run jls
  }
  const clientOptions = {
    documentSelector: ['python'], // run jls on py files
  }
  const client = new LanguageClient(
    'coc-jls', // the id
    'coc-jls', // the name of the language server
    serverOptions,
    clientOptions
  )
  context.subscriptions.push(services.registLanguageClient(client))
}

This code tells coc to register a new language server named coc-jls that executes the command jedi-language-server when Vim edits Python file(s). Now build the project with the following command:

yarn

Open an empty file (test.py works) with Vim and run the following command:

:set runtimepath^=~/src/sandbox/coc-jls

This command will help coc discover your extension. Depending on your coc configuration, you should see diagnostics, have autocompletion, etc. In the long term, you may want to manage your coc extension as a plugin or a Vim package, but we’ll stay local and simple for now.

User configuration¶

We’ve wrapped jedi-language-server with a coc extension and now it’s time to add some user configuration! Coc provides a simple mechanism to help users communicate their configuration values like VSCode: through the package.json. Make sure the following lines are in your package.json:

{
  "contributes": {
    "configuration": {
      "type": "object",
      "title": "coc-jls configuration",
      "properties": {
        "coc-jls.enabled": {
          "type": "boolean",
          "default": true,
          "description": "Enable coc-jls extension"
        }
      }
    }
  }
}

This says that the coc-jls extension contributes a configuration key coc-jls.enabled whose default value is true. If a user wants to disable coc-jls, they must add the following line to the top level of coc-settings.json:

{
  "coc-jls.enabled": false
}

Now we must add some logic to our typescript code to read the user configuration and disable the server if the user has chosen not to have it enabled:

import { ExtensionContext, services, workspace, LanguageClient } from 'coc.nvim'

export async function activate(context: ExtensionContext): Promise<void> {
  // BEGIN NEW CODE
  const config = workspace.getConfiguration('coc-jls')
  const isEnable = config.get<boolean>('enable', true)
  if (!isEnable) {
    return
  }
  // END NEW CODE
  const serverOptions = {
    command: 'jedi-language-server', // run jls
  }
  const clientOptions = {
    documentSelector: ['python'], // run jls on py files
  }
  const client = new LanguageClient(
    'coc-jls', // the id
    'coc-jls', // the name of the language server
    serverOptions,
    clientOptions
  )
  context.subscriptions.push(services.registLanguageClient(client))
}

Now run yarn and voilà: users can now disable coc-jls without needing to uninstall the extension. Success!

Deployment¶

To “deploy” this extension, one approach is to upload your project GitHub, add it to your Package or Plugin manager configuration, and install. If you’re using Vim-Plug, put the following in your vimrc:

Plug 'your-username/coc-jls', {'do': 'yarn install --frozen-lockfile && yarn build'}

And run:

:PlugInstall

This will download your Git repository, install all necessary dependencies, build your project, and make sure it’s automatically added to Vim’s runtime path for coc can discover the extension.

You can also do all this manually if you know what you’re doing.

Coc extensions can also be deployed to npm. This method is out of scope for this post, but you should be able to figure it out by referencing the coc-jedi codebase.

Wrapping up¶

At this point, you should have a working coc extension that wraps an executable language server. You may be wondering how you can add more features, automatically download and manage the executable for your users, and make your coc extension the most user friendly interface since the iPod click wheel. Worry not: these features are implemented in coc-jedi. Please refer to that codebase and ask any further questions in the comments below. And don’t feel obligated to restrict your wanderings to jedi-language-server: you can wrap any executable language server in a coc extension! Please use your powers for good.

Gitignore: two tricks

2018-12-02T00:00:00-05:00

Table of Contents

Ignore directory contents, keep directory
Ignore my weird files (in secret)
Conclusion

Most modern software developers are aware of Git and its accompanying gitignore file. This post offers two lesser-known-yet-useful techniques to ignore files from a Git-based version controlled project. The first technique enables Git to ignore every file in a directory but the directory itself. The second technique enables a user to automatically ignore a file without adding it to a project’s root gitignore.

Note: when discussing “gitignore” and “tern-project“, I may sometimes add a “.” before the filename. Whether I do or do not, I’m talking about the same file (eg, “.gitignore” == “gitignore”).

Ignore directory contents, keep directory¶

It is often convenient to have a directory whose contents are not committed to version control. For example, this directory may contain large non-general data files (images, financial data, etc). Additionally, these could be small files with sensitive information specific to one person (passwords, ssh keys, etc). Either way, we do not want these files committed to our project’s version control system. For the purposes of this conversation, we’ll call this folder “instance” and place it at the root of our project.

├── app
│   └── __init__.py
├── .gitignore          <-- Project gitignore
└── instance
    ├── big-file.jpg
    └── passwords.txt

A naive approach suggests that we add the instance folder to our project’s gitignore:

*.pyc
__pycache__/
# other ignored lines
instance/

Unfortunately, this proves problematic. What happens if our system relies on a file being in the instance folder at runtime? A user of our system would need to know to create the instance folder AND place a file there, forcing them to perform two manual steps.

A preferable solution: place a gitignore in the “instance” folder with special contents.

├── app
│   └── __init__.py
├── .gitignore          <-- Project gitignore
└── instance
    ├── big-file.jpg
    ├── .gitignore      <-- Special gitignore
    └── passwords.txt
-----------------------------------
special .gitignore contents:
# Ignore all files in this directory
# EXCEPT for this .gitignore file
*
!.gitignore

I find myself using this trick so often that I’ve added the instance folder and its accompanying gitignore file into my project skeletons.

# ~/.bashrc (I actually use zsh + ~/.zshrc, but it's the same)
function make_instance() {
  mkdir instance
  cat > instance/.gitignore <<EOL
# Ignore all files in this directory
# EXCEPT for this .gitignore file
*
!.gitignore
EOL
}

Finally, you may note that this technique does commit one file (the gitignore) to version control. IMHO, this is a small price to pay for elegance.

Ignore my weird files (in secret)¶

My development environment is highly customized. I’m an advanced user of Neovim, Tmux, Zsh, and other Unix-like utilities, and have set up a cross-language, bespoke IDE with a configuration unto itself. Benefits include software development bliss. One downside: my environment may require files that I should not commit to another person’s repository.

One example: when programming in Javascript, I rely on a Tern server for auto-completion in Neovim. Tern is configurable with a tern-project file which lets Tern know important things like the project’s ECMAScript version, which runtime environment it’s expecting, etc. These configurations differ by project, so it’s often necessary for me to place a customized tern-project file at the root of a Javascript project.

If I own a Javascript codebase and control its development, placing a tern-project file at my project’s root is a no-brainer. Not only can collaborators help me keep it up to date, but I can also influence more people to adopt Tern! Unfortunately, if I don’t own the codebase I’m working on, things become a bit trickier. I want to benefit from Tern but I don’t want my first contributions on a project to include my custom development environment configuration. I’d simply add this file to the project’s .gitignore file, but that would still be an me-specific contribution, so I’m left with two choices. Either I manually ignore tern-project every time I “git add/commit”, or I find a way to make Git do that work for me.

Luckily, Git has a built-in way for users to ignore their own weird files without needing to make any updates to a project’s version-controlled files. This method is mentioned in gitignore‘s documentation, but not many people are aware it exists.

Using your favorite editor, go to the root of any Git repository, and open the text file “.git/info/exclude”. It should look like this:

# Contents of PROJECT-ROOT/.git/info/exclude:
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~

Simply add your desired file patterns here and they will be ignored by you, and only you.

# Contents of PROJECT-ROOT/.git/info/exclude:
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
.tern-project

Once you’ve done this, I recommend submitting one or two impressive pull requests. You should be invited as a repository collaborator (or its equivalent) in no time. With credentials in hand, you can then suggest the group adopt your awesome Tern-based workflow (and your humble tern-project file) into the project. At this point, you should remove the “.tern-project” line from .git/info/exclude.

Conclusion¶

Ignoring directory contents while retaining directories is easy. Privately ignoring the files only you care about (for now) is also easy. Hopefully you learned something from this post and happy hacking!

Python function pipelines

2018-06-02T00:00:00-05:00

Table of Contents

The problem
Unix pipes
Example
Full script
Conclusion

If your Python code represents a function pipeline, it should look like a function pipeline. This post presents a simple, strongly-typed function pipeline for your personal projects to make beautiful, explicit, Unix-like pipelines in Python. Requires Python 3.6 or greater.

I was reading through the pytorch reinforcement learning documentation today and came across the following irksome pattern:

...

def forward(self, x):
    x = F.relu(self.bn1(self.conv1(x)))
    x = F.relu(self.bn2(self.conv2(x)))
    x = F.relu(self.bn3(self.conv3(x)))
    return self.head(x.view(x.size(0), -1))
..

The manipulation and reassignment of a variable x to itself as it is passed through a neural network’s gateways makes neural networks that much harder to follow. IMHO, explicit variable reassignment is almost always bad because it makes it hard for my limited human brain to track a variable name’s associated data during program execution. If a program tells me that x is an integer on line 10 and then reassigns it to a string on line 20, I will want to delete said program from my computer, take a cold shower, and exact revenge on the program’s author. With that established, let’s just avoid reassignment altogether.

The problem¶

In the above example, variable reassignment is convenient. It prevents the programmer from needing to come up with new names for each state in the connected pipeline. How can we find an elegant way of avoiding explicit variable reassignment in the pipeline use-case while still producing readable, performant code? The answer: take inspiration from Unix pipes!

Unix pipes¶

In the Unix shell, virtually everything we work with has the same type: a file. “Everything is a File” makes it easy to chain functions together because all commands take files as inputs and return files as outputs. One common way to chain system calls in Unix is called the anonymous pipe, which enables programmers to chain command line programs together to manipulate a text stream. See the following example:

#!/bin/bash

# Count the number of words
echo "hello hello hello world world" | wc -w

# Count the number of words that are not "hello"
echo "hello hello hello world world" | sed 's/hello//g' | wc -w

# Count the number of words that are not "world"
echo "hello hello hello world world" | sed 's/world//g' | wc -w

# Should print the following to console:
# 5
# 2
# 3

Notice that data flows through the pipeline and no variable reassignment is used. The Unix pipeline is beautiful in this regard; I’d like to build something similar in Python.

Example¶

Let’s say we have the following three functions:

def add_5(value: int) -> int:
    return value + 5

def add_6(value: int) -> int:
    return value + 6

def add_7(value: int) -> int:
    return value + 7

Our example problem: take an initial integer 0 and add 5, then 6, then 7 to it using our three functions so that our final result is 18.

Solution 1: name each step in the pipeline¶

We can do slightly better than the pytorch example and create a unique variable name for each step in the process.

x_0 = 0
x_5 = add_5(x_0)
x_6 = add_6(x_5)
x = add_7(x_6)

print(f'x={x}')

Besides being pretty ugly / hard to read, this creates some useless names in our module’s scope. Do we really need to give each step its own name? We don’t use the steps anywhere else and this makes the pipeline pretty hard to edit. This solution almost makes we wish we could go back to reassignment. Fortunately, the entire pipeline can be expressed with one name thanks to the “reduce” function.

Solution 2: use the “reduce” function¶

from functools import reduce
y = reduce(
    lambda value, function: function(value),
    (
        add_5,
        add_6,
        add_7,
    ),
    0,
)

print(f'y={y}')

This example uses the standard library “reduce” function. Originally, “reduce” was intended to take a list of values and collapse them into one value. This use-case is described well in the Python documentation and in this YouTube video. Here, we use it a bit differently. Instead of taking a list of values and applying a “collapsing” function to them, we take a list of functions and pass a value through them. This has obvious advantages over solution 1: we can easily swap new functions in and out of the pipeline to suit our needs without needing to adjust any other code / rename pipeline steps (because we haven’t named the steps at all!).

This solution frees us from naming each step in our pipeline but it does have some disadvantages: “reduce” is kind of hard to read and this use-case isn’t quite standard. Whenever we find ourselves using a standard library function in a confusing way, that’s a signal that we should probably define our own function to make this clearer to ourselves and to those who read our code.

Solution 3: custom “pipeline” function¶

from typing import TypeVar, Callable, Sequence

T = TypeVar('T')

def pipeline(
        value: T,
        function_pipeline: Sequence[Callable[[T], T]],
) -> T:
    '''A generic Unix-like pipeline

    :param value: the value you want to pass through a pipeline
    :param function_pipeline: an ordered list of functions that
        comprise your pipeline
    '''
    return reduce(lambda v, f: f(v), function_pipeline, value)

z = pipeline(
    value=0,
    function_pipeline=(
        add_5,
        add_6,
        add_7,
    )
)

print(f'z={z}')

This solution is elegant and explicit. It is generic and works with mypy. As long as our function pipeline contains only functions that take our value’s type and return our value’s type (similar to Unix command line utilities, where everything is a file), this pipeline will successfully pass our value in a type-safe way.

Full script¶

#!/usr/bin/env python
'''Simple function example'''

def add_5(value: int) -> int:
    return value + 5

def add_6(value: int) -> int:
    return value + 6

def add_7(value: int) -> int:
    return value + 7

x_0 = 0
x_5 = add_5(x_0)
x_6 = add_6(x_5)
x = add_7(x_6)

print(f'x={x}')

from functools import reduce
y = reduce(
    lambda value, function: function(value),
    (
        add_5,
        add_6,
        add_7,
    ),
    0,
)

print(f'y={y}')

from typing import TypeVar, Callable, Sequence

T = TypeVar('T')

def pipeline(
        value: T,
        function_pipeline: Sequence[Callable[[T], T]],
) -> T:
    '''A generic Unix-like pipeline

    :param value: the value you want to pass through a pipeline
    :param function_pipeline: an ordered list of functions that
        comprise your pipeline
    '''
    return reduce(lambda v, f: f(v), function_pipeline, value)

z = pipeline(
    value=0,
    function_pipeline=(
        add_5,
        add_6,
        add_7,
    )
)

print(f'z={z}')

Conclusion¶

If your system resembles a pipeline, don’t reassign your piped variable to itself. There is a better way and it’s pretty much built into Python. You’ll just need to care enough to use it.

Smaller python docker containers

2018-04-21T00:00:00-05:00

Table of Contents

The problem
Examples
Explanation
- Copying betwen Docker build stages in multi-stage build
- Difference between “pip install” and “pip wheel”
Conclusion
Special thanks

If your Docker Python build requires system dependencies that are NOT required at runtime, structure your build as follows:

Use a multi-stage build
Stage 1 installs system dependencies and uses them to build local wheels
Stage 2 begins from the same base as Stage 1, copies wheels from Stage 1, and installs the wheels
The rest of your build will be based on Stage 2

If you follow these steps, you’ll end up with the smallest-possible Python Docker container with all your Python dependencies intact.

Note: this post references Docker 18.03, Python 3.6, and pip 10. I assume that you are running CPython (Python’s reference implementation).

The problem¶

We want to do a Python system build using Docker. Python system builds often require installing third-party code. This third-party code may contain code or resources that must be compiled during their installation. For simplicity’s sake, assume we are talking about source code in the C programming language. Since a Docker container will be our “target machine”, we’ll need a C compiler in our Docker container. Unfortunately, C compilers are large programs. Since we plan to scale our number of containers up and down based on the demand for its provided service, the image should ideally be as small as possible.

Basically, we want to build C code with a C compiler and then throw away the C compiler to save space in our deployment image.

Examples¶

The following examples should clarify the problem and its resolution. Note: I’m assuming that you’re using a POSIX-inspired system.

Setup¶

Copy the following Makefile into your current working directory.

.PHONY: build-break
build-break:
    docker build -t blog-python:break -f ./Dockerfile.break .

.PHONY: build-big
build-big:
    docker build -t blog-python:big -f ./Dockerfile.big .
    docker images

.PHONY: build-uninstall-big
build-uninstall-big:
    docker build -t blog-python:big-uninstall -f ./Dockerfile.uninstall .
    docker images

.PHONY: build-small
build-small:
    docker build -t blog-python:small -f ./Dockerfile.small .
    docker images

Note: you might need to convert spaces to tabs.

Example 1: broken build requiring a C compiler¶

We have a simple, entrypoint-less Docker container in which we must install uWSGI. In the uWSGI quickstart guide, its developers clarify that it “is a (big) C application, so you need a C compiler (like gcc or clang) and the Python development headers”.

Copy the following code into a file called “Dockerfile.break”:

FROM python:3.6-alpine as breakimage

RUN pip install uwsgi

Now run the following shell command in the same directory as your Dockerfile.break.

$ make build-break

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-tkd8plx9/uwsgi/setup.py", line 137, in <module>
    'Programming Language :: Python :: 3.6',
  File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-install-tkd8plx9/uwsgi/setup.py", line 77, in run
    conf = uc.uConf(get_profile())
  File "/tmp/pip-install-tkd8plx9/uwsgi/uwsgiconfig.py", line 747, in __init__
    raise Exception("you need a C compiler to build uWSGI")
Exception: you need a C compiler to build uWSGI

At the end of our failed build, we see this Traceback (in addition to other helpful messages). Consistent with the uWSGI documentation, our system has said that we “need a C compiler to build uWSGI”. We’ll do that in example 2.

Example 2: large build with C compiler installed¶

In this example, we’ll install our system dependencies so uWSGI can actually be built.

Copy the following code into a file called “Dockerfile.big”:

FROM python:3.6-alpine as bigimage

RUN apk add --no-cache linux-headers g++

RUN pip install uwsgi

Now run the following shell command:

$ make build-big

REPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE
blog-python         big                 8a68d0dad407        Less than a second ago   251MB
python              3.6-alpine          8eb1c554687d        16 hours ago             90.4MB

In the “build-big” make target, I’ve included a command to list all Docker images on your system. Because of this command, you should see something close to the following in your terminal:

The good¶

The image built successfully.

The bad¶

The image is unnecessarily large.

We’re planning on scaling our web-service to handle a decent amount of traffic. Scaling will involve deploying many images on many servers. Larger images take longer to deploy and (obviously) take up more space than smaller images.

The ugly¶

We are including an unnecessary dependency.

We don’t need a C compiler in the image, so the C compiler is an unnecessary dependency. Including an unnecessary dependency in our runtime image is a horrible design, similar to including an unnecessary Python dependency in our requirements.txt or setup.py. As great software developers, we HATE bad system design, so let’s find a way to resolve the “bad” and the “ugly” while preserving the “good”!

Example 3: failed attempt at simply “uninstalling” C compiler¶

Unfortunately, if we want to reduce our image size, we cannot simply “uninstall” the C compiler. For reasons that I do not fully comprehend at this time, Docker caches anything you install in an image, so uninstalling a dependency does NOT reduce the image size.

Copy the following code into a file called “Dockerfile.uninstall”:

FROM python:3.6-alpine as bigimage-uninstalled

RUN apk add --no-cache linux-headers g++

RUN pip install uwsgi

RUN apk del linux-headers g++

Now run the following shell command:

$ make build-uninstall-big

REPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE
blog-python         big-uninstall       10a0eb5d42aa        Less than a second ago   251MB
blog-python         big                 8a68d0dad407        11 minutes ago           251MB
python              3.6-alpine          8eb1c554687d        16 hours ago             90.4MB

Our efforts at removing our C compiler proved futile. At this point, lesser developers would give up and assume we’ve reached the end of the road. But you, dear reader, are reading my blog, and I know you’re better than that! Let’s dig deeper and find an elegant way shrink our Docker image!

Example 4: small final build without C compiler¶

This final example results in a small image with uWSGI installed and without a C compiler. It relies heavily on multi-stage builds and on pip wheels.

Copy the following code into a file called “Dockerfile.small”:

###########################################
# Throwaway image with C compiler installed
FROM python:3.6-alpine as bigimage

# install the C compiler
RUN apk add --no-cache linux-headers g++

# instead of installing, create a wheel
RUN pip wheel --wheel-dir=/root/wheels uwsgi

###########################################
# Image WITHOUT C compiler but WITH uWSGI
FROM python:3.6-alpine as smallimage

COPY --from=bigimage /root/wheels /root/wheels

# Ignore the Python package index
# and look for archives in
# /root/wheels directory
RUN pip install \
      --no-index \
      --find-links=/root/wheels \
      uwsgi

Now run the following shell command:

$ make build-small

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
blog-python         small               b952f6280b00        1 second ago        97.4MB
<none>              <none>              91c7bb911f32        3 minutes ago       249MB
blog-python         big-uninstall       10a0eb5d42aa        23 minutes ago      251MB
blog-python         big                 8a68d0dad407        34 minutes ago      251MB
python              3.6-alpine          8eb1c554687d        16 hours ago        90.4MB

Notice that the image tagged “small” is ~61% smaller than its “big” counterparts. It has 7 additional MB from its base alpine container. These megabytes represent only the uWSGI library itself. We’ll need to make modifications to uWSGI itself to get any smaller. I leave uWSGI modifications as an exercise for the reader.

Explanation¶

Two key points are responsible for our Docker build’s success:

Reliance on copying between image stages in Docker multi-stage builds. This gets around caching problems with a single image
Understanding the difference between “pip install” and “pip wheel”

Copying betwen Docker build stages in multi-stage build¶

Unless we explicitly specify a target, Docker multi-stage builds will tag their last stage. Downstream build stages can reference upstream build stages and copy resources from them, similarly to how resources can be copied from any local or remote file system into a traditional Docker container. Therefore, we “compile” our Python code in one build stage and copy this compiled code in another build stage. Since the code no longer needs to be compiled, we don’t need to a C compiler or Linux headers. As the coup de grâce, our build’s final stage is not based on any image with a C compiler installed, so this approach completely avoids Docker’s caching complexities.

Thanks to Docker’s multi-stage builds, we are able to compile our Python package and avoid deploying the build’s system dependencies in our final image.

Difference between “pip install” and “pip wheel”¶

Docker multi-stage builds are cool and all, but I’ve seen many articles about them. Python’s packaging tool, pip, hasn’t gotten as much careful attention from the blogging community. Hopefully this section can clear up one common point of confusion: pip install vs pip wheel.

pip install¶

This is the command most people are familiar with. At a high level, it takes a Python package, runs its setup.py, downloads and installs its dependencies, and potentially does a lot more. Run “pip install” when you want to expand a package’s contents and use it as its author intended.

A good mental model: “pip install” takes a consolidated bundle of code / build instructions and places the package’s content and dependencies wherever they need to go on an operating system. Once “pip install” runs on our machine, file placement throughout our file system can be pretty hamajang, depending on a package’s setup.py instructions.

pip wheel¶

This tool is mostly used by library developers wanting to distribute their packages in a user-friendly way. For example, scikitlearn, a popular Python library for machine learning, requires a lot of system dependencies to build. Many Python users, especially data scientists, are either unwilling or unable to install these dependencies on their host machines. This user-characteristic led to unfortunate platforms like Anaconda (author opinion). On a more mature note, for those of us with the appropriate dependencies installed, the installation process would often take a very long time; C, FORTRAN, and possibly other languages each needed to be compiled, and installing code written in these languages often leads to a long coffee break.

Wheels enable Python developers to compile a package, and its dependencies, in a distributable form targeting common operating system architectures. Today, most scikitlearn users install it using its wheel, which takes a fraction of the time of the regular build process.

A good mental model: “pip wheel” takes a Python package, makes it ready to be installed on any target machine WITHOUT its build dependencies, and puts it in ONE easily-distributed archive file.

Why we care about this?¶

Not all Python packages are distributed as wheels. There are some packages, based mostly on C, that are hard to compile once and use in many places. uWSGI appears to be one of those packages. To build our final image, we construct a throw-away container to construct a wheel for uWSGI.

Conclusion¶

When building a Docker container for a Python application, we can install packages requiring build-time system dependencies AND remove these system dependencies from our final Docker image through a combination of Docker multi-stage builds, pip wheel, and pip install.

Special thanks¶

This post took inspiration from this post by Alejandro Guirao. I am indebted to Alejandro for publishing his creative use of docker multi-stage builds in the context of Python systems.

Vim line numbers

2018-04-07T00:00:00-05:00

Table of Contents

The Problem
Wrapping my own solution
The Game-Changing Plugin
Conclusion

If you want your Vim line numbers to be relative and/or not relative at the correct times, I recommend installing the myusuf3/numbers.vim plugin.

The Problem¶

When editing text in a Vim window, I use relative numbers to help me use motions across text relative to my cursor. However, when I have multiple windows open, relative numbers look pretty weird in windows that I am not currently editing. It would be nice for Vim to intelligently alternate between relativenumber and norelativenumber based on my Vim cursor location. Base Vim does not have this capability, so we have three options:

Accept a suboptimal workflow
Wrap our own solution in our .vimrc
Find a Plugin

I lived with option 1 for a while, but eventually grew too annoyed. I then tried researching plugins, but thought I understood the problem well-enough to write my own solution. So I went straight to option 2 and tried wrapping my own solution.

Wrapping my own solution¶

The following code represents my original solution:

function! ToggleRelativeNumber()
  if &rnu
    set norelativenumber
  else
    set relativenumber
  endif
endfunction

function! RNUInsertEnter()
  if &rnu
    let w:line_number_state = 'rnu'
    set norelativenumber
  else
    let w:line_number_state = 'nornu'
  endif
endfunction

function! RNUInsertLeave()
  if w:line_number_state == 'rnu'
    set relativenumber
  else
    set norelativenumber
    let w:line_number_state = 'nornu'
  endif
endfunction

function! RNUWinEnter()
  if exists('w:line_number_state')
    if w:line_number_state == 'rnu'
      set relativenumber
    else
      set norelativenumber
    endif
  else
    set relativenumber
    let w:line_number_state = 'rnu'
  endif
endfunction

function! RNUWinLeave()
  if &rnu
    let w:line_number_state = 'rnu'
  else
    let w:line_number_state = 'nornu'
  endif
  set norelativenumber
endfunction

" autocmd that will set up the w:created variable
autocmd VimEnter * autocmd WinEnter * let w:created=1
autocmd VimEnter * let w:created=1
set number relativenumber
augroup rnu_nu
  autocmd!
  "Initial window settings
  autocmd WinEnter * if !exists('w:created') |
        \setlocal number relativenumber |
        \endif
  autocmd User Startified setlocal number relativenumber
  " Don't have relative numbers during insert mode
  autocmd InsertEnter * :call RNUInsertEnter()
  autocmd InsertLeave * :call RNUInsertLeave()
  " Set and unset relative numbers when buffer is active
  autocmd WinEnter * :call RNUWinEnter()
  autocmd WinLeave * :call RNUWinLeave()
augroup end

The good¶

The solution worked for most windows and tabs, most of the time.

The bad¶

The code is a bit involved and it takes a little time to explain to others.

It relies on window-local variables (w:line_number_state, etc). These exacerbate Vim’s already-difficult state-management woes.
Several global functions are defined
There are some quirks I don’t fully understand around the creation of variables during Vim startup (hence lines 50 and 51).

Despite these mild downsides, I was pretty proud that the solution mostly worked. That is, until I wasn’t.

The back-breaking straw¶

My custom solution did not work appropriately with some of my plugins. Namely, it didn’t play well with majutsushi/tagbar, which I use frequently enough for this feature-dearth to become royally annoying. Therefore, after learning the ins-and-outs of window-specific variables and every Vim autocmd, I went back to the Plugin ecosystem to see if I’d missed anything…

The Game-Changing Plugin¶

Turns out a wonderful developer already solved this problem for me. Assuming you use junegunn/vim-plug to manage your plugins, place the following code in your .vimrc

call plug#begin('~/.vim/plugged')
" Relative Numbering
Plug 'myusuf3/numbers.vim'
" Put the rest of your plugins below...
call plug#end()

" Now, exclude the plugins you don't want numbers to deal with
let g:numbers_exclude = ['startify', 'gundo', 'vimshell']

This will give you a great editing experience. See below for a screencast:

Conclusion¶

Numbers.vim provides a usable line-numbering solution with minimal required configuration. I regret nothing about my bespoke journey, but I’m glad that Numbers.vim is my destination.