Building a personal knowledge base

2020-03-17

Table of contents

Recently I’ve set up a knowledge base to document any learnings or interesting resources I come across. It is accessible at https://kb.hbenjamin.com. In this post I describe my knowledge base workflow and the tooling I’ve built around it.

Requirements

On storage:
- Must be stored in plain text.
On reading:
- Must be readable in plain text.
- Must be viewable from a web browser on mobile or desktop.
On editing:
- Must be editable with a simple text editor.
- Must be editable from a web browser on desktop.
On discovery:
- Must be searchable from the terminal.
- Must be searchable from vim.
- May be searchable from the web browser. (Optional)

Design choices

Discovery: search over categorization

I focused on search over categorization as the primary discovery mechanism. While I do organize entries into sub-directories, I’ve opted not to do tagging for now. As the number of entries grow I might consider having a more thorough taxinomy.

File format

Each entry is written in markdown. Markdown is generally readable, moreso if using an editor with syntax highlighting. It being a plaintext file format means that it will always remain readable in the future.

Versioning

The knowledge base is versioned using git. The only concern I have is if I’ll have a need for binary files in the future. It’s plausible that I might want to store PDF files for instance. I’ll reconsider my storage choice if the number of binary files happens to grow large enough.

Web tech stack

Hugo as the static site generator: Each knowledge base entry is essentially a post.
Netlify for web hosting and automatic deployments.
NetlifyCMS for editing content through a web browser.

I’ve had good experiences using this stack for my personal site, so it makes sense to do the same for my knowledge base.

Directory structure and content

Here’s a snippet of my directory structure at this time:

content/
  climbing/
    _index.md
    trips.md
  tech/
    docker.md
    elasticsearch.md
    postgres.md
    pprof.md
  fitness.md
  travel.md

Here’s a snippet of what an entry might contain:

---
title: Climbing
---

## Injuries

- [Ester Smith on nagging finger injuries](https://eu.blackdiamondequipment.com/en_GB/esther-smith-nagging-finger-injuries/esther-smith-nagging-finger-injuries.html)
- [Steven Low - Overcoming Tendonitis](https://stevenlow.org/overcoming-tendonitis/)

## Training

- [Matt Pincus: Principles of Core Training](https://www.trainingbeta.com/matt-pincus-principles-of-core-training/)
- [Crimpd app](https://www.crimpd.com/). Climbing specific exercises and training routines, on and off the wall.

As you can see there’s nothing particularly special about how entries are represented. Plain markdown with optional YAML frontmatter. I can write and publish these posts from anywhere, including mobile.

Tooling

A command-line program to manage and navigate my knowledge base. Written in Go, the source can be found at github.com/benjaminheng/kb. kb requires fzf for fuzzy navigation.

Install using:

go get -u github.com/benjaminheng/kb

Browsing files

Invoking kb without any arguments spawns a fzf finder for jumping to a particular file. Typically this is the most common way I interact with my knowledge base. Selecting a file opens it in vim (or your choice of editor).

  pprof reference :: /Users/benh/dev/knowledge-base/tech/pprof.md
  PostgreSQL :: /Users/benh/dev/knowledge-base/tech/postgres.md
  Elasticsearch :: /Users/benh/dev/knowledge-base/tech/elasticsearch.md
  Docker :: /Users/benh/dev/knowledge-base/tech/docker.md
  Fitness :: /Users/benh/dev/knowledge-base/fitness.md
> Climbing :: /Users/benh/dev/knowledge-base/climbing.md
  6/6
>

Performing git operations

$ kb git --help
Performs git operations on the knowledge base

Usage:
  kb git [command]

Available Commands:
  pull        Pulls changes from remote
  push        Pushes changes to remote
  status      Shows working tree status

Since my knowledge base is versioned using Git, I wanted a way to quickly perform common Git operations without having to switch working directories and execute git commands manually. I’ve implemented three common operations: pull, push, and status.

The commands pull and status are fairly self-explanatory; the former invokes git pull and the latter git status.

However the push command is a composition of several git commands to make my life easier.

git ls-files --others --modified to check if there are files to commit. If none, then do git status and exit.
git add . to stage changes or new files.
git status to provide user feedback on what will be commited.
Prompt for user confirmation. If no confirmation, do git reset . and exit.
Prompt the user for a commit message. (default: “Update <filename>")
git commit -m "<commitmessage>" to create a commit.
git push to push changes upstream.

The end result is that after making a change to a file and invoking kb git push, the command will automatically push my changes upstream with minimal effort on my part.

Final thoughts

I’m quite happy with the solution I’ve ended up with. The knowledge base being available through a web browser is a surprisingly important feature. It being easily accessible has led to me updating it more often.

As for the tooling, it does what I want for the moment. My terminal workflow is to use kb to open the selected file in vim, then use my vim setup for further search and filters. It’s quite an efficient setup, given my familiarity with vim for code navigation.

Overall I had anticipated things being more complicated. However as I made progress, I found that many features I envisioned we simply not needed, or were simpler to implement that expected. Using Hugo, rendering the knowledge base into a website was a piece of cake. Netlify made the web deployment trivial, and NetlifyCMS took care of web editing. The tooling was simpler than expected as well; I found that I could delegate the bulk of code navigation to vim.