Stars are a popularity signal, not a procurement decision. Evaluating open source means checking whether the project solves your problem, can be operated safely, and will still be viable when you need upgrades in two years.
The typical failure looks like this: someone finds a project with an impressive README and forty thousand stars, demos it on sample data in an afternoon, and it enters production a month later. A year on, the sole maintainer has stepped away, a major upgrade breaks the data model, and nobody wrote down how to leave. None of that was hidden—it was all visible in the repo the day the decision was made. Nobody looked because the stars looked like an answer.
This checklist replaces that afternoon demo with a structured pass: start from the workflow you must support, then verify license, maintainership, security, docs, and a realistic proof of concept—not the README hero metrics. It takes days, not weeks, and it front-loads exactly the failures that are expensive to discover later.
Key takeaways
- Define the workflow and PoC success criteria before you open the repo.
- Read the LICENSE and dependency notices—not a blog summary.
- Recent releases and bus factor beat raw commit or star counts.
- Require a disclosure process, backup/restore path, and upgrade story.
- Price the exit: export formats and a plausible replacement path are part of the evaluation, not an afterthought.
- Reject criteria should be written down before the PoC starts.
1. Start with the problem, not the repo
- Write the workflow you must support in one paragraph.
- List must-have integrations and non-negotiable constraints (offline, SSO, air-gap).
- Define what “good enough” looks like for a two-week proof of concept.
- Note which catalog stacks you already operate so ops load stays realistic.
2. License and obligations
- Read the repository license and notice files—not a blog summary.
- Check dependency licenses that ship in your distribution.
- Involve counsel for distribution, embedded devices, or copyleft edge cases.
3. Maintainership and activity
- Recent releases matter more than raw commit counts—a project can be busy and still never ship.
- Look for bus factor: how many people can ship a release? Check who actually cut the last three.
- Skim issue response quality on bugs that resemble your use case, not the average across all issues.
- Check whether the project is company-backed, foundation-held, or volunteer-run—each fails differently when funding or interest shifts.
- Prefer clear governance over anonymous single-maintainer silence—unless the scope is tiny and forkable in a weekend.
4. Security and operations
- Is there a SECURITY.md or disclosure process?
- How are dependencies updated? Are container images signed or pinned?
- Can you back up and restore data with documented steps?
- What breaks during major version upgrades?
5. Documentation and escape hatches
Great docs reduce total cost of ownership. Demand export formats, APIs, and a plausible replacement path. If leaving requires a custom archaeology project, price that risk now.
A quick heuristic for documentation quality: try to answer three operational questions using only the docs—how do I back up and restore, how do I upgrade between major versions, and how do I integrate authentication? If any answer requires reading source code or Discord archaeology, that is the support experience your team will have at 2 a.m. during an incident. Docs written only for the happy-path install are one of the most reliable predictors of expensive operations later, because they reveal what the maintainers have and have not had to think about.
Red flags and green flags
- Green: releases with changelogs and migration notes; maintainers who say “no” to features with reasons; a SECURITY.md with a real contact; tests that run in CI on every PR.
- Green: multiple people cutting releases, boring governance documents, and issues closed with explanations rather than silence.
- Red: the last release predates the last hundred commits by a year—activity without shipping usually means nobody can cut a release.
- Red: the roadmap exists only in a company's sales deck; community PRs sit unreviewed while vendor-priority features merge weekly.
- Red: breaking changes land without migration notes, or version numbers move without semantic meaning.
- Red: security reports handled in public issues, or a history of CVEs fixed silently without advisories.
6. Run a realistic PoC
- Import real sample data, not demo fixtures only—edge cases in your data are the point of the exercise.
- Test the failure case: restore from backup, revoke credentials, rotate secrets. A PoC that only tests the happy path proves nothing.
- Perform one version upgrade during the PoC if releases allow it—upgrade friction is the cost you will pay most often.
- Measure operator time for install, upgrade, and monitoring, and multiply by your realistic staffing—not by the enthusiasm of the person running the PoC.
- Write down why you would reject the project—then see if those risks are acceptable to the people who will own it.
Common evaluation mistakes
- Letting the person most excited about the tool run the evaluation alone—enthusiasm is useful energy and a terrible measurement instrument. Pair them with the person who will be on call for it.
- Evaluating the demo dataset instead of your data: demo fixtures are curated to work. Your fifteen-year-old CSV exports with mixed encodings are the actual test.
- Comparing the candidate against a perfect imagined alternative rather than against the incumbent's real, known flaws—new tools get judged on promises, old tools on incidents, and that asymmetry distorts decisions.
- Skipping the upgrade test because the PoC version is current: upgrade friction is the recurring cost, and a PoC that never upgrades measures the one thing you will do least.
- Treating community size as maintenance quality: a huge Discord with unanswered operational questions is a worse signal than a small issue tracker with precise answers.
- Sunk-cost adoption: three weeks into a PoC that keeps disappointing, the pressure is to adopt anyway because the time is spent. The written reject criteria from step one exist precisely to make stopping a defensible decision.
Frequently asked questions
- Are GitHub stars enough to evaluate a project?
- No. Stars measure attention at some point in history—many heavily starred projects are effectively abandoned, and many excellent niche tools never trend. Procurement needs problem fit, license, maintainership, security process, docs, and a PoC on your data. Use stars only as a weak tiebreaker between otherwise equal candidates, never as the reason to shortlist.
- What license checks matter most?
- Three layers: the repository LICENSE itself, notices for dependencies you actually ship, and any recent relicensing history—a project that changed terms once is more likely to change them again. Involve counsel when you distribute binaries, embed in devices, or face copyleft obligations; skip the lawyer for internal-only tooling under permissive licenses. Always read the file, not a blog summary of it.
- How do I judge maintainership?
- Look at who cut the last three releases and how far apart they were—that answers both cadence and bus factor in five minutes. Then read issue threads for bugs resembling your use case: a maintainer who triages clearly beats one who merges fast but never responds. Finally, identify the funding model (company, foundation, volunteers), because each fails differently. Prefer readable governance over anonymous silence unless the project is small enough to fork and maintain yourself.
- What should a two-week PoC prove?
- That the project survives contact with your reality, not the demo's. Concretely: real data imports cleanly, backup and restore actually work, credentials can be rotated, one version upgrade goes through without archaeology, and the operator time is something your team can sustain. End the PoC by re-reading the reject criteria you wrote before starting—if the risks you named are now acceptable, adopt; if you find yourself rationalizing them away, that is the signal to stop.
- Where do I go next after evaluation?
- Use catalog guides and stack pages for ops context, then decide self-host vs hosted with a clear owner, RTO/RPO, and upgrade cadence.
Conclusion
Treat open-source adoption like any other dependency decision: fit, license, maintainers, security, and exit. The checklist above is deliberately front-loaded—license and maintainership problems are cheap to find in an hour of reading and expensive to discover in production. Use catalog signals as context, then verify with your own PoC on your own data, and keep the written reject criteria as the record of what you accepted. Two years from now, when an upgrade or a maintainer change forces a decision, that document is what turns a surprise into a plan.
Build a stack for this use case.
Answer nine practical questions and compare three transparent architectures with costs, free limits, lock-in, and migration paths.
Build my stack
