Study Projects
How to Read a Project
Do not start at the top of the file tree. A large repository read from the first file teaches nothing. Use a method:
- Pick one question. "How does it evict?" "What happens on a hash collision?" "How is a lookup kept O(log n)?" One question gives you a path through the code.
- Read the tests first. Tests show the public behavior and the edge cases the authors were afraid of. The
_test.gofile next to a structure is the best documentation it has. - Read the API with
go doc.go doc github.com/hashicorp/golang-lru/v2prints the exported names and comments without opening an editor. - Find the core type and its invariants. Every structure has one: "the list is ordered by recency", "keys in a node are sorted", "every level has half the nodes". Find where each method preserves it.
- Run it. Clone it, run the tests (
go test ./...) and the benchmarks (go test -bench . -benchmem). Change one line and see which test fails. - Compare with your own version. After a lesson, write the structure from the lesson's demo, then read the library's. The differences (locks, generics, pooling, corner cases) are the lesson.
- Read the history.
git log -pandgit blameon a tricky line lead to the issue and the reasoning behind it; bug-fix commits show what went wrong in practice.
Algorithm Collections
The best places to start: many structures in one repository, small files, readable code.
| Project | What to study | Prepare with |
|---|---|---|
| TheAlgorithms/Go | A large community collection of algorithms in Go: sorting, searching, graphs, strings, math, dynamic programming, data structures. Compare its versions with the demos here; it is also the friendliest place for a first contribution. | Sorting, Graph Algorithms, Dynamic Programming |
| emirpasic/gods | "Go Data Structures": lists, stacks, maps, sets, trees (AVL, red-black, B-tree), heaps, all behind common interfaces. Study how one container interface serves many implementations. | Data Structures, Trees |
| Workiva/go-datastructures | Production-oriented structures: augmented and interval trees, queues, a Fibonacci heap, futures. Notice how concurrent access is handled. | Advanced Trees, Production Structures |
| zyedidia/generic | Generic (type-parameter) containers for Go: a compact codebase to learn how generics change the shape of a data-structure library. | Arrays and Slices, Hash Tables |
| gonum/gonum | Numerical computing for Go. Read the graph packages for shortest paths, spanning trees and flow written for real use, then compare with your own Dijkstra. | Graphs, Graph Algorithms |
| dominikbraun/graph | A small, generic graph library: directed and undirected graphs, topological sort, shortest paths, cycle detection. A good size for reading a whole library in an evening. | Graphs |
The Go Standard Library
The standard library is the most carefully reviewed Go code there is, and it is part of the golang/go repository, so it costs nothing to read. Browse it on pkg.go.dev (each function has a "source" link) or in the src directory of the repository.
| Package | What to study | Prepare with |
|---|---|---|
container/heap, container/list, container/ring | Short and complete: a heap over any sort.Interface-like type, a doubly linked list with a sentinel (the design used in the LRU demo), a circular list. | Heaps, Linked Lists |
| sort and slices | A real sort is a hybrid: pattern-defeating quicksort with insertion sort for small ranges and a heap-sort fallback to keep the worst case O(n log n). Also generic binary search. | Sorting, Searching |
the built-in map (runtime/map*.go) | The runtime's hash table: how buckets or groups are laid out, how it grows, how it randomizes iteration order. Read after writing your own chained table. | Hash Tables |
| sync and sync/atomic | sync.Map, sync.Pool, sync.Once: small pieces of code with detailed comments on why they are designed as they are. | Concurrent Structures |
strings, bytes, regexp | Substring search that picks a strategy by pattern length (including Rabin-Karp), and a regular-expression engine built on automata rather than backtracking. | String Algorithms |
| golang.org/x/time/rate | The standard token-bucket limiter. It computes tokens from elapsed time, like the demo, and takes a clock so that it can be tested. | Rate Limiters |
| golang.org/x/sync | singleflight (call collapsing), errgroup and semaphore: the production versions of the patterns in the concurrency demo. | Call Collapsing |
Caches
| Project | What to study | Prepare with |
|---|---|---|
| hashicorp/golang-lru | LRU, 2Q and ARC caches. The simplelru package is the map-plus-list design from the lesson; the outer package adds locking. A short path from the demo to production code. | LRU Cache |
| dgraph-io/ristretto | A high-throughput cache with a TinyLFU admission policy that uses a Count-Min sketch and a Bloom filter: the fix for the scan weakness, built from structures in the Advanced Techniques lesson. The TinyLFU paper is short and readable. | Advanced Techniques |
| maypok86/otter | A modern, high-performance cache with state-of-the-art eviction and lock-free-style reads. Read its README benchmarks and design notes to see how caches are compared. | Production Structures |
| allegro/bigcache | A sharded cache that keeps entries in large byte slices so that the garbage collector has almost no pointers to scan. Shows how memory layout, not the algorithm, can be the bottleneck. | Arrays and Slices |
Storage Engines and Indexes
Databases are where data-structure theory meets disks. These engines are written in Go, are used in serious systems, and are small enough to read.
| Project | What to study | Prepare with |
|---|---|---|
| etcd-io/bbolt | An embedded key-value store built on a copy-on-write B+tree in a memory-mapped file. One of the most readable database cores in Go: pages, cursors, transactions. | Trees, Advanced Trees |
| cockroachdb/pebble | A log-structured merge-tree (LSM) engine in the LevelDB/RocksDB family: a skip list in memory, sorted immutable files on disk, compaction, Bloom filters to skip files. | Sorting, Bloom Filters |
| dgraph-io/badger | Another LSM store that keeps large values in a separate log so that compaction moves only keys. Compare the design choices with Pebble's. | Hash Tables, Heaps |
| etcd-io/etcd | The consistent store behind Kubernetes: a replicated log (Raft), a versioned key-value index and a B-tree of revisions. Large; read one package, such as the in-memory index, not the whole thing. | Advanced Trees |
| google/btree and tidwall/btree | In-memory B-trees with iteration, ranges and cheap copy-on-write clones. Compare the two: generics, API, and how each splits and merges nodes. | Trees |
| blevesearch/bleve | Full-text search: text analysis, inverted indexes, scoring. Shows how tries and sorted term lists become a search engine. | String Algorithms |
Compact and Probabilistic Structures
| Project | What to study | Prepare with |
|---|---|---|
| bits-and-blooms/bitset and bloom | A bitset over uint64 words and a Bloom filter on top of it: the same design as the demos, with sizing helpers and serialization. | Bit Manipulation |
| RoaringBitmap/roaring | Compressed bitmaps that pick a representation (sorted array, bitmap, or runs) per block of 65,536 values. A lesson in adapting the structure to the data. | A Bitset |
| axiomhq/hyperloglog | A HyperLogLog with a sparse form for small counts, so that a sketch with a few items does not cost 16 KB. Compare its estimator with the demo. The original Flajolet et al. paper is the reference. | HyperLogLog |
| cespare/xxhash | A fast non-cryptographic hash in Go with an assembly version. Hashes like this feed Bloom filters, sketches and hash rings; see why quality and speed both matter. | Hash Tables |
Distributed Systems and Infrastructure
| Project | What to study | Prepare with |
|---|---|---|
| buraksezer/consistent | Consistent hashing with bounded loads: a ring that also caps how much any server receives. Compare with the ring in the demo. For a different idea, read the jump consistent hash paper (a dozen lines of code). | Consistent Hashing |
| kubernetes/kubernetes | Very large, but its work queue (client-go's util/workqueue) is a self-contained study: de-duplicating queue, delayed queue and rate-limited retries with exponential backoff. | Stacks and Queues, Heaps |
| prometheus/prometheus | Its time-series database (tsdb) combines an inverted index over labels with compressed sample blocks. A study in indexing and compact encoding under a heavy write load. | Hash Tables, Advanced Techniques |
A Reading Plan
Pair each lesson with one reading task and one small experiment:
| After lesson | Read | Then |
|---|---|---|
| Heaps | container/heap | Rewrite your heap to satisfy heap.Interface and use it for Dijkstra. |
| Sorting | the slices sort in the Go source | Find where it switches to insertion sort and benchmark that threshold on your machine. |
| Hash Tables | the built-in map's runtime code | Explain why map iteration order is randomized, and test it. |
| Graph Algorithms | gonum/graph or dominikbraun/graph | Compare its shortest path with your own on the same random graph. |
| Advanced Trees | google/btree, then bbolt | Find the node split, and check how the tree stays balanced. |
| Advanced Techniques | bits-and-blooms/bloom, roaring | Measure the false-positive rate of the library against its formula. |
| Production Structures | golang-lru, x/time/rate, singleflight | Replace your demo's structure with the library and rerun the demo's checks against it. |
Contributing
Contributing is how reading becomes skill: someone who knows the project reviews your code. Start small; the first goal is a merged change, not an impressive one.
A Ladder of Contributions
- Use the project. Build something with it. Every confusing moment is a documentation bug.
- Read the contribution guide. Look for
CONTRIBUTING.md, the issue templates and the code-of-conduct file before writing any code. - Fix the small things. Typos and unclear comments, a missing example in the docs, a missing test case for an edge case.
- Add tests and benchmarks. A test that compares a structure with a slow oracle on random input (the technique used throughout this roadmap) is a valuable contribution to almost any library. So is a fuzz test (Go fuzzing tutorial) or a benchmark for a hot path.
- Reproduce a bug. Open the issue list, filter by labels such as
good first issueorhelp wantedwhere the project uses them, and comment before you start so that two people do not do the same work. A failing test that reproduces the bug is already half of the fix. - Add or improve an algorithm. In a collection such as TheAlgorithms/Go, check first that the algorithm is not already there, then follow the repository's conventions for layout, documentation and tests.
- Take on a larger change. Discuss the design in an issue first. Large unannounced pull requests are the ones most often declined.
Before You Send a Change
- Run
gofmt -l .,go vet ./...andgo test ./..., and fix what they report. Add-racefor anything concurrent. - Keep the change small and about one thing, with a clear title and a description that says what was wrong and how you checked the fix.
- Match the surrounding style, comment density and naming, even where you would have chosen differently.
- Expect review comments. Answer each one, push follow-up commits, and be patient: maintainers are often volunteers.
- Do not paste code you cannot explain, and do not submit code whose license you do not know.
Contributing to Go Itself
The Go project has its own process: changes are reviewed on Gerrit, not through GitHub pull requests, and contributors sign a contributor license agreement. The contribution guide explains the steps. Documentation, examples and tests in the standard library are a realistic place to begin; the x/ repositories, such as golang.org/x/sync and golang.org/x/time, follow the same process.
Practice Lab
- Diff of two LRUs. Read
golang-lru/simplelruand 73_lru_cache.go. List every difference, and say for each whether it is about correctness, speed, or convenience. - Oracle for a library. Pick a container from
emirpasic/godsand write a random test that checks it against a plain slice or map. Report what you find, even if the answer is that everything agrees. - Limiter comparison. Replay the traffic from 74_rate_limiters.go through
golang.org/x/time/rate. How many requests does it accept in the boundary test, and why? - Read a B-tree. In
google/btree, find where a full node is split. Draw the before and after for a node of 3 items, and compare with your notes from Advanced Trees. - First contribution. Choose one project from this page and write down: the issue or gap you will address, the file you will change, and the test that will prove it. Submit it.
More material is on the References page. The Demo Examples list has every runnable program in this roadmap.