TL;DR

Built 17k Go CLI tools as static binaries using Zig cross-compilation. Install with soar install package-name instead of go install. No Go toolchain required. 67% success rate, 1:10:10 ratio of modules to executables.

No more dumpster diving through GitHub repos to find quality CLI tools. Pre-built static binaries that work everywhere, instantly.

FAQ

Q: Why not just use go install?
A: If you're a Go developer, go install works great for you. This is for end users who want CLI tools without installing Go - sysadmins, DevOps folks, or anyone who just wants tools to work.

Q: What about package managers like apt/homebrew?
A: Static binaries work everywhere without dependency hell. One binary runs on any Linux distro, any version, without requiring specific library versions.

Q: Is this secure? You're redistributing random code.
A: All builds run in isolated GitHub Actions with public logs. Every binary has build provenance, attestations, and checksums. You can verify exactly what was built and how.

Q: Why Zig specifically?
A: Go's built-in cross-compilation is excellent for standard static binaries, but we specifically need static PIE (Position Independent Executable) binaries. When using -buildmode=pie on glibc systems like GitHub Actions, Go produces dynamically linked executables and generates static linking warnings. Zig elegantly solves this by providing musl libc, which natively supports static PIE binaries without the glibc complications - giving us the security benefits of PIE while maintaining true static linking. See: https://github.com/golang/go/issues/64875, https://gitlab.alpinelinux.org/alpine/aports/-/issues/15809, https://bugs.gentoo.org/924632

Q: What happens when builds fail?
A: Build logs are public, and we're considering auto-filing issues upstream to help maintainers fix cross-compilation problems.

Intro

Pkgforge hosts the world's largest collection of prebuilt, static binaries that work everywhere without dependencies. While our main repos include hand-picked packages and manually maintained by our team, we had an ambitious idea: what if we could automatically harvest CLI tools from ecosystems like Go's module registry, build them as static binaries using Zig's powerful cross-compilation capabilities, and made them available to everyone?

Instead of manually curating every package, we decided to tap into existing package ecosystems and automate the entire process. After two weeks of intensive development and countless iterations, we made this idea a reality.

We also recommend reading our previous post: Cross-Compiling 10,000+ Rust CLI Crates Statically, as that contains some background/context which will be relevant later in this post.

Ingesting Go Modules

When we started this project, we assumed Go would have something resembling Rust's crates.io—a proper package registry with APIs, metadata, and basic ecosystem tooling. We discovered the reality was quite different.

Our journey began with what seemed like the obvious approach: scraping pkg.go.dev. After all, this is Google's official Go package discovery site, beautifully displaying exactly the metadata we needed—descriptions, repository information, and package statistics. The site had everything we wanted in a gorgeous, human-readable format, so surely they'd provide an API for programmatic access?

Unfortunately, this approach proved unreliable. We spent considerable time building scrapers, only to encounter frequent failures as the site's responses changed or received 403 errors when our requests appeared automated.

What made this particularly maddening was seeing all the data we needed displayed right there on the website, but completely inaccessible through any official API. There's been a GitHub issue requesting this basic functionality since 2020, and it remains stubbornly ignored by the Go team.

It was only after this frustrating waste of time that we discovered the truth buried in pkg.go.dev's about page: they source their data from the Go Module Index. This, however highlights a fundamental difference in design philosophy: while Rust makes their registry API prominent and easily accessible, Go's decentralized approach means finding this info takes some digging.

Trials & Tribulations

Once we found the Module Index, we were forced to develop a multi-pronged strategy combining several data sources:

• The Go Module Index (the buried treasure we should have found on day one)

• GitHub's API for repositories with Go CLI tags and topics (because the Module Index provides zero metadata about package purpose)

• Community-curated lists of Go CLI utilities (maintained by volunteers because Google won't)

• Scraping deps.dev (Google's other package service that actually works)

• Web scraping pkg.go.dev when all else failed (since they still won't provide an API)

For comparision, contrast this with what we had to do in our previous blog: https://blog.pkgforge.dev/cross-compiling-10000-rust-cli-crates-statically#heading-ingesting-cratesio: we can simply download their complete database dump, query it locally, and have instant access to comprehensive metadata for every package. No scraping, no downloading source code, no heuristic analysis—just clean, structured data that respects developers' time and computational resources.

This revealed some challenges with Go's minimalist design philosophy when it comes to ecosystem tooling. Unlike Rust's ecosystem where Cargo.toml explicitly declares binaries, categories, keywords, and descriptions, Go's "simplicity-first" approach created a nightmare of discovery.

Rust doesn't just excel at manifest design—they actually care about their ecosystem. They provide complete database dumps and have thoughtful policies like RFC 3463 that genuinely support developers and researchers. Meanwhile, Go provides:

• No central registry like crates.io

• No standardized metadata format

• No proper package manager (just go get which is barely more than a glorified downloader)

• No way to programmatically determine what a package actually does without downloading and analyzing its source code

Go's module system makes it stupidly hard to answer basic questions like 'what does this package do?’. While Rust has cargo search, cargo info, and rich metadata queries, Go developers are stuck with a system that can't even tell you what a package does without downloading it first. There's no package statistics, no trending packages, no search functionality beyond basic text matching—essentially none of the features that make a package ecosystem usable at scale.

These architectural decisions led us to build a sophisticated toolchain of four separate utilities to answer questions like "Is this package a CLI tool?"

• go-indexer: Fetches modules list from the Module Index

  

• go-enricher: The digital scavenger that enriches our sparse dataset by scraping deps.dev (ironically, also built by Google), GitHub's API, and whatever other third-party services we can find— An absurd workaround that highlights the dysfunction of Go's registry. While deps.dev shows Google clearly understands the problem and how to solve it, they continue to maintain a "minimalist" module system that offers virtually none of this functionality natively, refusing to integrate such capabilities into official tooling.

  

• filter-urls: Removes unreachable download URLs to prevent build failures—because of course the "decentralized" system has reliability issues.

  

• go-detector: The crown jewel of Go's registry dysfunction: a system that downloads, extracts, and analyzes tens of thousands of Go modules with complex code analysis and scoring algorithms—just to determine if a package is a CLI tool. We're burning gigabytes of bandwidth and compute parsing ASTs and import patterns to answer a question that should be a single field in a manifest.

  

So while the Go team clearly understands the value of package metadata (they built an entire website around it), they've deliberately chosen not to make it programmatically accessible, forcing every tooling developer to become a digital archaeologist. A well-designed ecosystem would provide comprehensive search, rich metadata, usage stats, trending insights, integrated tooling, full workflows for initialization, publishing, testing, and documentation, along with open data access for researchers and developers. Instead, Go gives us go get—basically git clone with extra steps—and calls it a day.

After building an entire toolchain out of necessity and automating it with GitHub Actions, we now generate everything ourselves—but it shouldn't have taken four tools and industrial-scale source analysis to answer basic questions about Go packages. All of this could have been avoided if Go simply offered what other ecosystems have for years: proper registries with real APIs, metadata, and tooling support. Go's 'keep it simple' philosophy created a Rube Goldberg machine for package discovery, while other ecosystems prove that developer-friendly design is both possible and actively maintained.

Package Selection

$ go-indexer --start-date "2024-01-01" --output "./INDEX.jsonl" --verbose
Go Modules Index Fetcher
┌────────────────────────────────────────────────────────────┐
│ Configuration                                              │
├────────────────────────────────────────────────────────────┤
│ Start date                   :                  2024-01-01 │
│ End date                     :                  2025-07-01 │
│ Output file                  :               ./INDEX.jsonl │
│ Max concurrent               :                          30 │
│ Max retries                  :                           3 │
│ Batch size                   :                        2000 │
│ Timeout                      :                         30s │
│ Dry run                      :                       false │
│ Resume mode                  :                       false │
│ Process output               :                        true │
└────────────────────────────────────────────────────────────┘
📅 Generated 547 dates to process
📂 Temp directory: /tmp/.tmpWhWALv
📦 Combining daily files into final output...
📋 Combining [████████████████████████████████████████] 547/547 files ✓ 18518000 total lines                                                                                                 🔄 Post-processing output file...
🔍 Processing ✓ Processed 18518000 lines                                                                                                                                                     📊 Grouping 1515437 unique sources...
💾 Writing processed output...
✅ Post-processing complete!
📄 Processed file: ./INDEX.processed.json
📊 Unique sources: 1515437
🏷️  Total versions: 17982265

🎉 Processing Complete!
┌────────────────────────────────────────────────────────────┐
│ Final Statistics                                           │
├────────────────────────────────────────────────────────────┤
│ Duration                     :                     233.10s │
│ Days processed               :                     547/547 │
│ Total records                :                    18518000 │
│ Errors                       :                           0 │
│ Rate                         :           79444 records/sec │
└────────────────────────────────────────────────────────────┘
📊 Final output: ./INDEX.jsonl (2352.59 MB, 18518000 lines)

Since we ended up with ~ 1.5 Million Go modules, we needed to set some constraints & filter for what we actually wanted to build:

1. Must be indexed by index.golang.org within the last year i.e > 2024-01-01

2. Must have proper description, home page etc. from our go-enricher

3. Must be a CLI & pass our go-detector.

4. Must have a reasonable number of GitHub stars (> 2) or downloads to indicate community usage

🐹 Total Unique Modules (2024+):   ████████████████████████████████████████████████████████████ 1,515,437
    ↓
☑️ Selected/Eligible Modules:      ████████████████████████████████████████▌                    516,694
    ↓
🧬 Enriched with deps.dev API:     ███████████████████████████████████▌                         331,333
    ↓
🔗 Filtered for cached URLs:       ████████████████████████████▌                                250,391
    ↓
🔩 Filtered for CLI packages:      ███████████▌                                                 72,496
    ↓
🔧 Filtered for Popularity:        █████▌                                                       17,039

✅ Modules to Build: 17,039
// Total unique modules whose metadata was scraped/fetched: 1,515,437 (> date -d 'last year' '+%Y-01-01')  
// Total modules that matched the selection criteria: 516,694
// Total modules enriched with deps.dev API: 331,333
// Total modules that are actually cached: 250,391
// Total modules detected as CLI: 72,496
// Total modules to build: 17,039

We ended up with ~ 17,000 Go modules that we now planned to compile.

Build Tool

With over 17,000 modules to build on individual GitHub Actions runners, speed was paramount. While Go offers excellent built-in cross-compilation, we needed to use musl for pure static linking & also to avoid things like:

warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Packages trying to link with external system libraries via CGO (other than ibc) will fail by design, as we target pure Go implementations.

Our heavy docker images used for official packages consumed 2-3 minutes just for pulling and extraction, making them unsuitable for this scale. We needed a solution that was minimal & worked out of the box.

Enter Zig: The C/C++ cross-compiler toolchain, which provides:

• Zero setup cross-compilation: No need for target-specific toolchains

• Universal C/C++ compiler: Single installation handles all our target architectures

• Fast compilation: Minimal overhead compared to traditional cross-compilation setups or containers

• MUSL for Static linking : Perfect for our portable binary goals

We also used jpeddicord/askalono to automatically detect & copy over licenses.

Build Constraints

To achieve truly portable, optimized, and statically linked relocatable binaries, we applied the following comprehensive build constraints using CGO with Zig as our cross-compiler:

#Build Environment
export CGO_ENABLED=1
export CGO_CFLAGS="-O2 -flto=auto -fPIE -fpie -static -w -pipe"
export CC="zig cc -target ${target_triplet}"
export CXX="zig c++ -target ${target_triplet}"
export GOOS=linux
export GOARCH=${TARGET_ARCH}

#Go Build Flags
go build -a -v -x -trimpath \
         -buildmode="pie" \
         -buildvcs="false" \
         -ldflags="-s -w -buildid= -linkmode=external" \
         -tags 'netgo,osusergo' \
         -extldflags="-s -w -static-pie -Wl,--build-id=none"

1. Static PIE (Position Independent Executable): -buildmode=pie with -static-pie creates relocatable static binaries

2. CGO Enabled: CGO_ENABLED=1 allows interaction with C libraries (Core only) while maintaining static linking

3. Optimized CGO: CGO_CFLAGS=-O2 -flto=auto -fPIE -fpie -static -w -pipe enables LTO and position independence

4. Pure Go Networking: -tags 'netgo' uses Go's native network stack instead of libc

5. Pure Go User/Group: -tags 'osusergo' avoids libc for user/group lookups

6. Zig Cross-Compilation: Uses Zig as the C/C++ compiler for seamless cross-compilation with static linking

7. Fully Stripped: -trimpath -buildvcs=false -buildid= remove all debug info, symbols, and build IDs

8. No System Libraries: CGO is used only for the zig linker, not arbitrary system C library.

#These modules would error out in the following manner
error: could not find system library required by CGO
error while loading shared libraries: nameofthelib.so.1: cannot open shared object file: No such file or directory

Build Targets

While Soar supports any Unix-based Distro, due to lack of CI support for other Unix Kernel on GitHub Runners (natively, not VMs), we are limited to Linux only. We further refined our target matrix by excluding architectures approaching end-of-life:

Build Security

We are aware of supply chain security concerns in package ecosystems, so we wanted this to be as secure as our official repositories, by ensuring:

• Modules are downloaded from official Go proxy servers, like the official Go toolchain does

• CI/CD run on GitHub Actions, with temporary, scoped tokens per package

• Build Logs are viewable using: soar log ${PKG_NAME}

• Build Src is downloadable by downloading: {GHCR_PKG}-srcbuild-${BUILD_ID}

• Artifact Attestation & Build Provenance are created/updated per build

• Checksums are generated (& verified at install time by Soar) for each & every artifact per build

These measures ensure that even if a malicious module attempts to compromise the system, its impact is isolated and cannot affect other modules' integrity.

Build Workflow

17,000 multiplied by 4 targets, meant we would need to run ~ 70,000 instances of CI & also handle metadata, sanity checks, uploading to ghcr, all at the same time. We also set up a discord webhook to stream real-time progress updates to our discord server.

graph TD
  A[🐹 Go Module Registry] -->|📊 Scrape Metadata 🧬| B[🔍 Filter & Queue ⏳]
  B --> C[🏗️ GitHub Actions Matrix 🔄]
  C --> D[⚡ Zig Cross Compiler ⚒️]
  D --> E[📦 Static Binaries 🛠️]
  E --> F[🗄️ GHCR Registry ⬆️]
  F -->|📊 Generate Metadata 🧬| B
  F --> G[🚀 Soar ⬇️]

Key Insights and Findings

Build Success vs. Failure

At the time of writing (2025-07-01), we have queued ~ 12000 out of the total ~ 17000 modules. We approached this project with optimistic expectations but encountered a sobering reality.

🏗️ Build Pipeline by Success Rate
─────────────────────────────────────────────────────────────────────────
✅ Queued    ████████████████████████████████████████████████████████████ 11,866 (100.0%)
⚙️ Built     ██████████████████████████████████▌                          8,007 (67.5%)
❌ Failed    ████████████████████▌                                        3,859 (32.5%)
─────────────────────────────────────────────────────────────────────────

So what went wrong? We sampled about 100 of these error logs & concluded:

1. CGO Dependencies with System Libraries: The majority of failures stemmed from modules requiring CGO with system libraries that couldn't be statically linked or weren't available in our build environment

2. PIE Build Mode Compatibility: Some modules or their dependencies don't support Position Independent Executable (PIE) build mode

3. Platform-Specific Code: Many modules include platform-specific code or build constraints that fail during cross-compilation to newer architectures

#These typically fail cross-compilation
Modules that:
- Require dynamic system libraries that can't be statically linked
- Don't support PIE (Position Independent Executable) build mode
- Include assembly code for specific architectures
- Use CGO with complex system library dependencies
- Rely on runtime feature detection that conflicts with static PIE

##Examples

# CGO dependency failures
error: could not find system library 'libgtk-3' required by CGO
ld: library not found for -lssl

# PIE compatibility issues  
relocation R_X86_64_32 against symbol `main.version' can not be used when making a PIE object
linkmode external not supported on linux/riscv64

# Architecture-specific failures
undefined: syscall.SYS_EPOLL_CREATE1
build constraints exclude all Go files in package

Despite Go's excellent cross-compilation story, CGO dependencies with system libraries and PIE build mode compatibility remain the primary obstacles to universal static PIE compilation. This reinforces our strategy of targeting CLI tools that can be fully statically linked with minimal external dependencies.

Modules vs Executables

Another interesting insight from building at scale: many Go modules produce multiple executables. The ~ 8,000 modules we built successfully generated ~ 80,000 individual executables (Also referred to as binaries or packages)

🏗️ Build Pipeline by Executables
───────────────────────────────────────────────────────────────────────────────
📦 Modules Built       ████                                                       8,007 (100.0%) #Base Line
⚙️ Total Executables   ████████████████████████████████████████████████████████   80,082 (10x)
────────────────────────────────────────────────────────────────────────────────

This 1:10:10 ratio reveals how rich the Go CLI ecosystem actually is, with many projects providing comprehensive toolsuites rather than single utilities.

Native vs Cross

🏗️ Build Pipeline by Architecture
─────────────────────────────────────────────────────────────────────────────────
🖥️  x86_64-Linux      ████████████████████████████████████████████████████████ 20,423 (100.00%) #Base Line
🖥️  aarch64-Linux     ███████████████████████████████████████████████████████  20,025 (98.05%)
🖥️  riscv64-Linux     ███████████████████████████████████████████████████████  20,167 (98.75%)
🖥️  loongarch64-Linux ██████████████████████████████████████████████████████   19,467 (95.32%)
─────────────────────────────────────────────────────────────────────────────────

The consistent success rates across architectures demonstrate Go's excellent cross-platform story, with Zig providing seamless CGO cross-compilation. Newer architectures like loongarch64 show slightly lower compatibility rates, suggesting that architecture-specific code assumptions remain common in the ecosystem.

An interesting anomaly: riscv64-Linux has more executables than aarch64-Linux. This discrepancy occurs because some modules successfully build for non-standard targets like loongarch64-Linux and riscv64-Linux but fail for standard architectures due to build hooks and conditional compilation that trigger differently across targets.

You can explore detailed per-target build results here: PKGS_BUILT.json

CI Performance Metrics

Our primary build workflow (matrix_builds.yaml) handles the bulk of compilation, with additional workflows managing metadata and miscellaneous tasks. As we implement incremental builds (only rebuilding updated modules) and caching strategies, these metrics will improve significantly.

Average build time was ~ 2.5 minutes (slower than Rust because we try to build all packages with main).

Review

Compilation vs. Prebuilt Distribution

Compilation will always be slower than fetching prebuilt binaries, but the degree varies significantly based on module complexity and dependency count. For our demonstration, we'll use lazygit as a representative example, - results will vary significantly with more complex, dependency-heavy modules.

Note: We're not measuring CPU, disk, memory, or bandwidth usage here—try it yourself to experience the full performance difference.

Go Install

$ time go install -v "github.com/jesseduffield/lazygit@latest"
real    0m20.392
user    1m9.520s
sys     0m14.904s

Soar

#Soar uses pkg_ids to ensure exact match because we have too many packages
$ time soar install "lazygit#github.com_jesseduffield_lazygit:pkgforge-go"
real    0m4.371s
user    0m0.181s
sys     0m0.152s

Go's native tooling provides:

• Seamless integration with the Go ecosystem

• Automatic dependency resolution

• Built-in cross-compilation support

• Direct access to latest versions

Soar is different. We're not trying to replace go install for developers—we're making CLI tools accessible to everyone who just wants stuff to work without installing Go.

Conclusion

What started as let's build everything turned into a deep dive on Go's ecosystem quirks and why Zig is genuinely magical for cross-compilation.

Key Discoveries and Implications

The Go CLI ecosystem is remarkably mature and productive. Our 1:10:10 ratio of executables to modules reveals that the community is building comprehensive toolsuites with multiple utilities per project. This multiplier effect means that successfully building even a subset of available modules provides exponentially more value to end users.

Zig as a cross-compilation toolchain with CGO is transformative. By using Zig as our C/C++ compiler with CGO enabled, we eliminated the traditional complexity of cross-compilation toolchain setup while maintaining the ability to statically link libC dependencies and create PIE binaries. Zig’s toolchain is magic.

Static PIE linking remains both powerful and challenging. The ability to produce truly portable, relocatable binaries that work across any Linux distribution without dependencies is transformative for CLI tool distribution. However, achieving this with PIE (Position Independent Executable) mode requires careful consideration of dependencies and build strategies, even with Go's excellent standard library.

Broader Ecosystem Implications

Our work demonstrates that automated, large-scale binary distribution can significantly improve developer and end-user experience. The time savings—from over 20 seconds of compilation time to under 5 seconds of download time—represent meaningful productivity improvements.

More importantly, this approach democratizes access to Go CLI tools. Users no longer need Go installed, don't need to understand module paths, and can avoid compilation wait times. They can simply install and use tools, lowering the barrier to entry for adopting Go-based CLI utilities.

Future Roadmap

The pkgforge-go project will likely see these additions/improvements in the near future:

• Automated updates: Rebuild modules when new versions are published (this is partially implemented)

• Integration with Go toolchain: Maybe something similar to go install (specify a module path directly) but with prebuilt binaries

• Build optimization: Optimize CI Build times & reduce failures through better CGO handling

• Contribute Upstream: Opt-in system to automatically create GitHub issues with build logs when module compilation fails, helping maintainers improve cross-compilation compatibility

• Community Feedback: Listen to our users & the community to improve this project & hope for a widespread adoption beyond Soar

As we continue to refine and expand this system, we're excited about its potential to influence how the broader software community thinks about binary distribution. The combination of Go's compilation speed, Zig's cross-compilation capabilities, and automated CI/CD represents a powerful model for other ecosystems.

The ultimate goal is to create a world where installing and using CLI tools is as simple as possible, regardless of the underlying programming language or system dependencies. This project represents a significant step toward that vision, leveraging the best aspects of Go's compilation speed, Zig's cross-compilation capabilities, and static PIE linking to create truly portable, high-performance, relocatable CLI tools.

We invite the community to engage with this work, contribute improvements, and help us build a more accessible and efficient software distribution ecosystem. Together, we can make powerful CLI tools available to everyone, everywhere, without the traditional barriers of compilation and dependency management.

Links:

• Soar: https://github.com/pkgforge/soar

• Pkgforge-Go: https://github.com/pkgforge-go/builder

• Pkgforge-Discord: https://discord.gg/djJUs48Zbu

Pkgforge hosts the world's largest collection of prebuilt, static binaries that work everywhere without dependencies. While our main repos include hand-picked packages and manually maintained by our team, we had an ambitious idea: what if we could automatically harvest CLI tools from ecosystems like Rust's crates.io, build them as static binaries, and made them available to everyone?

Instead of manually curating every package, we decided to tap into existing package ecosystems and automate the entire process. After two weeks of intensive development and countless iterations, we made this idea a reality.

Ingesting Crates.io

Crates.io provides api access for individual crate lookups and bulk operations. Initially, our script iterated through the first 1,000 pages (sorted by downloads) with 100 crates per page, yielding approximately 111,000 crates. However, we soon encountered a significant bottleneck: we needed to query each crate individually to determine if it belonged to the command-line-utilities, or produced executables, i.e. contained [[bin]] in their manifest.

This approach proved impractical as we quickly hit rate limits and potentially violated the Usage Policy. Fortunately, RFC-3463 came to our rescue. Crates.io provides periodic database dumps at https://static.crates.io/db-dump.tar.gz . We quickly drafted a nifty cli using dtolnay/db-dump

Then it was just a matter of parsing this with a bit of jq, & automating it via GitHub Actions. Our workflow now generates all the data we will need, automatically.

Crate Selection

Since we ended up with over 111,000 crates, we needed to set some constraints & filter for what we actually wanted to build:

1. Should either be of category command-line-utilities: categories = ["command-line-utilities"]

2. Or must have a [[bin]] section in the Manifest.

3. Must be updated within the last year, i.e > 2024-01-01

🦀 Total crates fetched:            ██████████████████████████████████████████████████ 111,722
     ↓
📦 Crates with binary targets:      ████████████████████████████████  24,658
     ↓
🔧 Crates with CLI category:        ██████████░░░░░░░░░░░░░░░░░░░░░░   8,451
     ↓
📆 Recently updated (2024+):        ███████████░░░░░░░░░░░░░░░░░░░░░  10,033
     ↓ + ↑
✅ Crates to Build: 10,033

// Total crates whose metadata was scraped/fetched: 111,722  
// Total crates with [[bin]] in their Cargo.toml manifest: 24,658  
// Total crates with command-line-utilities category in their Cargo.toml manifest: 8,451  
// After filtering for outdated crates (> date -d 'last year' '+%Y-01-01')  
// Total crates to build: 10,033

We ended up with ~ 10,000 crates that we now planned to compile.

Build Constraints

To achieve truly portable, optimized, and statically linked binaries, we applied the following comprehensive build constraints:

#RUSTFLAGS
[+] Flags: -C target-feature=+crt-static \
           -C default-linker-libraries=yes \
           -C link-self-contained=yes \
           -C prefer-dynamic=no \
           -C embed-bitcode=yes \
           -C lto=yes \
           -C opt-level=3 \
           -C debuginfo=none \
           -C strip=symbols \
           -C link-arg=-Wl,-S \
           -C link-arg=-Wl,--build-id=none \
           -C link-arg=-Wl,--discard-all \
           -C link-arg=-Wl,--strip-all

1. Statically Linked: -C target-feature=+crt-static

2. Self Contained: -C link-self-contained=yes

3. All Features: --all-features

4. Link Time Optimization: -C lto=yes

5. All Optimizations: -C opt-level=3

6. Stripped: -C debuginfo=none -C strip=symbols

7. No System Libraries: Crates with system library dependencies will fail by design, as we target pure Rust implementations

   #These crates would error out in the following manner
   error: could not find system library 'openssl' required by the 'openssl-sys' crate
   error: Could not find system library 'sqlite3'
   error: pkg-config has not been configured to support cross-compilation
   

Build Tool

With over 10,000 crates to build on individual GitHub Actions runners, speed was paramount. While Cargo offers cross compilation features, it requires significant setup overhead. We needed a solution that worked out of the box.

Our heavy docker images used for official packages consumed 2-3 minutes just for pulling and extraction, making them unsuitable for this scale. This left us with rust-cross/cargo-zigbuild & cross-rs/cross. After some local testing, we decided to use Cross as it supported all the targets we needed & worked as advertised: “Zero setup” cross compilation

We also used jpeddicord/askalono to automatically detect & copy over licenses.

#The CMD Looks like
cross +nightly build --target "${RUST_TARGET}" -Z unstable-options \
     --all-features \
     --artifact-dir="${C_ARTIFACT_DIR}" \
     --jobs="$(($(nproc)+1))" \
     --release \
     --verbose

Build Targets

While Soar supports any \Unix-based Distro*, due to lack of CI support for other Unix Kernel on GitHub Runners (natively, not VMs), we are limited to Linux only. We further refined our target matrix by excluding architectures approaching end-of-life:

Build Security

We are aware of issues like https://github.com/rust-lang/cargo/issues/13897, so we wanted this to be as secure as our official repositories, by ensuring:

• Crates are downloaded from crates.io, like the official Cargo does.

• CI/CD run on GitHub Actions, with temporary, scoped tokens per package

• Build Logs are viewable using: soar log ${PKG_NAME}

• Build Src is downloadable by downloading: {GHCR_PKG}-srcbuild-${BUILD_ID}

• Artifact Attestation & Build Provenance are created/updated per build.

• Checksums are generated (& verified at install time by Soar) for each & every artifact per build.

These measures ensure that even if a malicious crate attempts to compromise the system, its impact is isolated and cannot affect other crates' integrity.

Build Workflow

10,000 multiplied by 4 targets, meant we would need to run ~ 40,000 instances of CI & also handle metadata, sanity checks, uploading to ghcr, all at the same time. We also set up a discord webhook to stream real-time progress updates to our discord server.

graph TD
  A[🦀 Crates.io API] -->|📊 Scrape Metadata 🧬| B[🔍 Filter & Queue ⏳]
  B --> C[🏗️ GitHub Actions Matrix 🔄]
  C --> D[⚙️ Cross Compiler ⚒️]
  D --> E[📦 Static Binaries 🛠️]
  E --> F[🗄️ GHCR Registry ⬆️]
  F -->|📊 Generate Metadata 🧬| B
  F --> G[🚀 Soar ⬇️]

Key Insights and Findings

Build Success vs. Failure

We approached this project with optimistic expectations but encountered a sobering reality. Out of approximately 10,000 crates queued for building:

🏗️ Build Pipeline by Success Rate
────────────────────────────────────────────────────────────────────────
✅ Queued    ████████████████████████████████████████    10,033 (100.0%)
⚙️ Built     ███████████████████████████                  5,779 (57.60%)
❌ Failed    ████████████                                 4,254 (42.40%)
────────────────────────────────────────────────────────────────────────

So what went wrong? We sampled about 100 of these error logs & concluded:

1. System Library Dependencies: The majority of failures stemmed from crates requiring system libraries that weren't available in our static build environment

2. Custom Build Systems: Many crates include build.rs files that fail when specified dependencies aren't met or when detecting system features during cross-compilation

    #These typically fail cross-compilation
    build.rs files that:
    - Detect system features
    - Link against system libraries
    - Generate code based on target environment
   

Despite years of Rust ecosystem maturation, system library dependencies remain the primary obstacle to universal static compilation. This reinforces our strategy of targeting CLI tools that can be fully statically linked.

Crates vs Executables

Another interesting insight from building at scale: many crates produce multiple executables. The ~ 5,800 crates we attempted generated ~ 21,000 individual executables (Also referred to as binaries or packages)

🏗️ Build Pipeline by Executables
──────────────────────────────────────────────────────────────────────────────────
📦 Crates Built       ██████████                                   5,779 (100.0%) #Base Line
⚙️ Total Executables  ████████████████████████████████████████     21,042 (364.0%)
──────────────────────────────────────────────────────────────────────────────────

This 3.6:1 ratio reveals how rich the Rust CLI ecosystem actually is.

Native vs Cross

🏗️ Build Pipeline by Architecture
─────────────────────────────────────────────────────────────────────────────────
🖥️  x86_64-Linux     ████████████████████████████████████████     5,627 (100.00%) #Base Line
🖥️  aarch64-Linux    ███████████████████████████████████████▌     5,586 (99.30%)
🖥️  riscv64-Linux    ██████████████████████████████████████▎       5,370 (95.40%)
🖥️ loongarch64-Linux ███████████████████████████████▋              4,459 (79.20%)
─────────────────────────────────────────────────────────────────────────────────

The consistent success rates across architectures demonstrate Rust's excellent cross-platform story, though newer architectures like loongarch64 show slightly lower compatibility rates. This suggests that architecture-specific code assumptions remain common in the ecosystem.

An interesting anomaly: Despite building 5,779 crates successfully, x86_64-Linux only shows 5,627 executables. This discrepancy occurs because some crates successfully build for non-standard targets like loongarch64-Linux and riscv64-Linux but fail for standard architectures due to build hooks and scripts that trigger differently across targets.

You can explore detailed per-target build results here: CRATES_BUILT.json

CI Performance Metrics

Our primary build workflow (matrix_builds.yaml) handles the bulk of compilation, with additional workflows managing metadata and miscellaneous tasks. As we implement incremental builds (only rebuilding updated crates) and caching strategies, these metrics will improve significantly.

Average build time was ~ 2 minutes.

Review

Compilation vs. Prebuilt Distribution

Compilation will always be slower than fetching prebuilt binaries, but the degree varies significantly based on crate complexity and dependency count. For our demonstration, we'll use fd-find as a representative example, though your experience may vary with more dependency-heavy crates.

Note: We're not measuring CPU, disk, memory, or bandwidth usage here—try it yourself to experience the full performance difference.

Cargo

$ time cargo install fd-find

real    0m51.893s
user    3m36.568s
sys     0m24.411s

Cargo Binstall/Quick Install

Cargo Binstall leverages prebuilt binaries, though it requires time for crate resolution: related issue

$ time cargo binstall fd-find --no-confirm

real    0m6.001s
user    0m0.118s
sys     0m0.083s

Cargo-Binstall and Cargo-Quickinstall are excellent tools that:

• Integrate with cargo install workflow

• Handle development dependencies and features

• Target developers who want faster cargo install

Soar takes a different approach:

• Distribution-focused: Static executables for end users

• No development integration: Not meant for cargo workflows

• Dependency-free: Zero system library requirements

• Cross-distribution: Works on any *nix system (MUSL/GLIBC)

Soar

#Soar uses pkg_ids to ensure exact match because we have too many packages
$ time soar install "fd#pkgforge-cargo.fd-find.stable:pkgforge-cargo"

real    0m1.695s
user    0m0.062s
sys     0m0.090s

Conclusion

This project represents more than just a build farm; it's a proof of concept & also a reality check for the whole ecosystem.

Key Discoveries and Implications

The Rust CLI ecosystem is remarkably rich and diverse. Our 3.6:1 ratio of executables to crates reveals that the community is building comprehensive toolsuites rather than single-purpose utilities. This multiplier effect means that successfully building even a subset of available crates provides exponentially more value to end users.

Cross-compilation compatibility has room for improvement. While Rust's cross-platform story is generally excellent, our 42.4% failure rate highlights that system library dependencies and architecture-specific assumptions remain significant obstacles. This suggests opportunities for the community to develop more portable alternatives to system library bindings.

Static linking is both powerful and challenging. The ability to produce truly portable binaries that work across any Linux distribution without dependencies is transformative for CLI tool distribution. However, achieving this requires careful consideration of build flags, dependencies, and compilation strategies.

Broader Ecosystem Implications

Our work demonstrates that automated, large-scale binary distribution is not only feasible but can provide significant value to the developer community. The time savings alone—from nearly a minute of compilation time to under two seconds of download time—represent a meaningful improvement in developer productivity.

More importantly, this approach democratizes access to CLI tools. Users no longer need to have Rust installed, understand compilation flags, or debug dependency issues. They can simply install and use tools, lowering the barrier to entry for adopting Rust-based CLI utilities.

Future Roadmap

The pkgforge-cargo project will likely see these additions/improvements in the near future:

• Automated updates: Rebuild crates when new versions are published (this is partially implemented)

• Integration with Cargo: Maybe something similar to what `cargo binstall` does.

• Build optimization: Optimize CI Build times & reduce Failures

• Contribute Upstream: Opt-in system to automatically create GitHub issues with build logs when crate compilation fails, helping maintainers improve cross-compilation compatibility

• Community Feedback: Listen to our users & the community to improve this project & hope for a widespread adoption beyond Soar.

As we continue to refine and expand this system, we're excited about its potential to influence how the broader software community thinks about binary distribution. The lessons learned here apply beyond Rust to any compiled language ecosystem, and we're eager to explore applications in Go, Zig, and other emerging systems languages. (Help us if you can)

The ultimate goal is to create a world where installing and using CLI tools is as simple as possible, regardless of the underlying programming language or system dependencies. This project represents a significant step toward that vision, and we're committed to continued innovation in this space.

We invite the community to engage with this work, contribute improvements, and help us build a more accessible and efficient software distribution ecosystem. Together, we can make powerful CLI tools available to everyone, everywhere, without the traditional barriers of compilation and dependency management.

Links:

• Soar: https://github.com/pkgforge/soar

• Pkgforge-Cargo: https://github.com/pkgforge-cargo/builder

• Pkgforge-Discord: https://discord.gg/djJUs48Zbu

Getting Soar onto systems with no download tools is a chicken-and-egg problem. Fresh containers, minimal distributions, and legacy Unix systems often lack curl, wget, or even working SSL libraries. You need a tool to get tools.

The Challenge

Consider these scenarios:

# Fresh Debian container
$ docker run -it debian:latest
# No curl, no wget, no ca-certificates

# Legacy system
$ uname -a
# SunOS legacy 5.10 Generic_147441-01 i86pc i386 i86pc
$ which curl wget
# command not found

# No Privileges
$ apt install curl
Error: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
Error: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

You need to download Soar, but you can't download Soar without download tools. Traditional solutions fail:

• apt install curl requires privileges and a working package manager

• Copying files requires existing access/infrastructure or is just too slow over scp

• Building from source needs compilers and dependencies

The Solution

Every system with bash has /dev/tcp - a built-in feature for raw TCP connections.

No external dependencies, no SSL requirements, just basic HTTP over TCP:

#From: https://github.com/hackerschoice/thc-tips-tricks-hacks-cheat-sheet?tab=readme-ov-file#4vii-file-download-without-curl
burl() {
    IFS=/ read -r proto x host query <<<"$1"
    exec 3<>"/dev/tcp/${host}/${PORT:-80}"
    echo -en "GET /${query} HTTP/1.0\r\nHost: ${host}\r\n\r\n" >&3

    #Skip headers until we hit the empty line
    while read -r line <&3; do
        # Remove carriage return and check if line is empty
        line=${line%$'\r'}
        [[ -z "$line" ]] && break
    done

    #Output only the body
    cat <&3
    exec 3>&-
}

#Examples:
burl "http://http.pkgforge.dev/https://ip.pkgforge.dev/json"
burl "http://http.pkgforge.dev/https://github.com/pkgforge/soar/releases/latest/download/soar-x86_64-linux" > "./soar"

This works on any bash system, but GitHub serves HTTPS. Legacy systems can't connect to HTTPS endpoints.

This works in:

• Fresh Docker containers (debian:latest)

• Legacy Unix systems without SSL

• Minimal environments with only bash

• Systems with broken certificate stores

The HTTP-to-HTTPS Proxy Bridge

The solution: an HTTP proxy that handles HTTPS termination. A Cloudflare Worker accepts HTTP requests and forwards them to HTTPS destinations:

Client (HTTP) → Worker (HTTP) → Target (HTTPS) → Worker → Client

// Extract target URL from path
const destUrl = url.pathname.substring(1);

// Forward to HTTPS endpoint
const response = await fetch(destUrl, {
  method: request.method,
  redirect: 'follow',
  headers: cleanHeaders(request.headers)
});

return response;

Edge Cases

The code of worker resides @https://github.com/pkgforge-dev/reverse-proxies/blob/main/http.pkgforge.dev/worker.js & handles several edge cases for compatibility:

#From a system that has curl, you can read the help page
$ curl -qfsSL "http://http.pkgforge.dev/help"
'
This is a reverse proxy that works over HTTP (also HTTPS) to proxy to HTTPS sites.
For example, if you are on an ancient system with no SSL and only bash,
you can use this to download a static curl binary & join the modern internet.

Simply append your URL to http://http.pkgforge.dev/$URL
Example, http://http.pkgforge.dev/https://example.com --> Requests example.com (HTTPS) --> Returns the Response Back
If you want to use the DNS or IP Address to access this proxy, then visit the endpoints below.
Each endpoint will print a template HTTP request (with bash /dev/tcp example).

If your client has DNS, visit: http.pkgforge.dev/self-dns
If your client has IPv4, visit: http.pkgforge.dev/self-ipv4
If your client has IPv6, visit: http.pkgforge.dev/self-ipv6

Source Code: https://github.com/pkgforge-dev/reverse-proxies/tree/main/http.pkgforge.dev
Contact: https://docs.pkgforge.dev/contact/chat
'

DNS-less clients: Provides IPv4/IPv6 endpoints for systems without DNS resolution:

# Use IP directly if DNS is broken
burl "http://172.67.74.226/https://github.com/..."

Header management: Strips problematic headers that interfere with proxying:

// Clean headers for forwarding
requestInit.headers.delete('host');
requestInit.headers.delete('connection');

Redirect handling: Follows redirects automatically since GitHub releases often redirect to CDN URLs.

Real-World Usage

This is used in Soar's official installer:

# Fallback for systems without download tools
if ! command -v curl >/dev/null && ! command -v wget >/dev/null; then
  raw_http_get "http://http.pkgforge.dev/https://github.com/pkgforge/soar/releases/latest/download/soar-${ARCH}" > "./soar"
  sed "1,/^\r\{0,1\}$/d" -i "./soar"
fi

Why This Matters

Soar needs to run on diverse systems - from cutting-edge containers to decades-old Unix boxes. Traditional package managers assume modern tooling exists. This bootstrap method ensures Soar can install itself anywhere bash runs.

Once Soar is installed, it handles all future downloads with proper SSL, certificate validation, and modern HTTP features. The proxy is just for the initial bootstrap.

Security Trade-offs

The proxy breaks end-to-end encryption for the bootstrap download. This is acceptable because:

• Soar releases are public binaries

• Checksums can be verified after download

• It's only used when no secure alternatives exist

• The alternative is manual file copying or no access at all

For production usage, Soar uses proper HTTPS connections.

Beyond Soar

This technique works for any static asset like webpages or files that needs to reach legacy systems:

# Get any GitHub release
burl "http://http.pkgforge.dev/https://github.com/user/repo/releases/latest/download/binary-linux-amd64"

# Public package repositories
burl "http://http.pkgforge.dev/https://packages.example.com/tool.tar.gz"

The proxy solves the fundamental bootstrap problem: getting the first tool onto a system so it can get better tools.

The complete proxy runs as a Cloudflare Worker, handling the HTTPS complexity while presenting a simple HTTP interface to legacy clients. Source code and deployment instructions are available in the reverse-proxies repository.

This architectural approach prioritizes compatibility over security for the narrow use case of system bootstrapping, enabling modern tools to run on systems that predate modern web infrastructure.

References & Further Reading

• https://rednafi.com/misc/http_requests_via_dev_tcp/

• https://github.com/fijimunkii/bash-dev-tcp

• https://github.com/hackerschoice/thc-tips-tricks-hacks-cheat-sheet

• https://www.gnu.org/software///bash/manual/bash.html

• https://tldp.org/LDP/abs/html/devref1.html

• https://ivanmosquera.net/2024/08/27/network-tools-inside-a-pod-exploring-dev-tcp-and-busybox/?utm_source=chatgpt.com