package main import ( "net/http" "path" "slices" "strconv" "strings" "sync" ) type language struct { Name string Colour string Bytes int64 Percent float64 } var languages = map[string][2]string{ ".go": {"Go", "#00add8"}, ".ml": {"OCaml", "#ef7a08"}, ".mli": {"OCaml", "#ef7a08"}, ".mly": {"OCaml", "#ef7a08"}, ".mll": {"OCaml", "#ef7a08"}, ".thy": {"Isabelle", "#fefe00"}, ".ML": {"Standard ML", "#dc566d"}, ".sml": {"Standard ML", "#dc566d"}, ".v": {"Rocq", "#d0b68c"}, ".lean": {"Lean", "#4a4a8c"}, ".agda": {"Agda", "#315665"}, ".hs": {"Haskell", "#5e5086"}, ".idr": {"Idris", "#b30000"}, ".c": {"C", "#555555"}, ".h": {"C", "#555555"}, ".cc": {"C++", "#f34b7d"}, ".cpp": {"C++", "#f34b7d"}, ".hpp": {"C++", "#f34b7d"}, ".rs": {"Rust", "#dea584"}, ".zig": {"Zig", "#ec915c"}, ".S": {"Assembly", "#6e4c13"}, ".s": {"Assembly", "#6e4c13"}, ".asm": {"Assembly", "#6e4c13"}, ".HC": {"HolyC", "#ffefaf"}, ".smt2": {"SMT", "#7b8d9a"}, ".py": {"Python", "#3572a5"}, ".rb": {"Ruby", "#701516"}, ".js": {"JavaScript", "#f1e05a"}, ".mjs": {"JavaScript", "#f1e05a"}, ".ts": {"TypeScript", "#3178c6"}, ".tsx": {"TypeScript", "#3178c6"}, ".java": {"Java", "#b07219"}, ".kt": {"Kotlin", "#a97bff"}, ".swift": {"Swift", "#f05138"}, ".scala": {"Scala", "#c22d40"}, ".ex": {"Elixir", "#6e4a7e"}, ".exs": {"Elixir", "#6e4a7e"}, ".erl": {"Erlang", "#b83998"}, ".gleam": {"Gleam", "#ffaff3"}, ".lua": {"Lua", "#000080"}, ".nix": {"Nix", "#7e7eff"}, ".sh": {"Shell", "#89e051"}, ".tex": {"TeX", "#3d6117"}, ".html": {"HTML", "#e34c26"}, ".css": {"CSS", "#663399"}, ".scm": {"Scheme", "#1e4aec"}, ".rkt": {"Racket", "#3c5caa"}, ".lisp": {"Common Lisp", "#3fb68b"}, ".clj": {"Clojure", "#db5855"}, ".fs": {"F#", "#b845fc"}, ".nim": {"Nim", "#ffc200"}, } var vendorDirs = []string{"vendor/", "node_modules/", "third_party/", "thirdparty/", "external/", "deps/", "dist/", "katex/"} func vendored(name string) bool { base := path.Base(name) if strings.Contains(base, ".min.") || strings.HasSuffix(base, ".pb.go") || strings.HasSuffix(base, "_generated.go") { return true } for _, d := range vendorDirs { if strings.HasPrefix(name, d) || strings.Contains(name, "/"+d) { return true } } return false } type langEntry struct { head string langs []language } var langCache = struct { sync.Mutex m map[int64]langEntry }{m: map[int64]langEntry{}} func (s *Server) repoLanguages(repo Repo) []language { g := Git{Dir: s.repoDir(repo)} head, err := g.Resolve("HEAD") if err != nil { return nil } langCache.Lock() e, ok := langCache.m[repo.ID] langCache.Unlock() if ok && e.head == head { return e.langs } out, err := g.run(nil, "ls-tree", "-r", "-l", "-z", head) if err != nil { return nil } bytes := map[string]int64{} var total int64 for _, rec := range strings.Split(string(out), "\x00") { meta, name, ok := strings.Cut(rec, "\t") f := strings.Fields(meta) if !ok || len(f) != 4 || f[1] != "blob" || vendored(name) { continue } l, known := languages[path.Ext(name)] if !known { l, known = languages[strings.ToLower(path.Ext(name))] } if !known { continue } size, _ := strconv.ParseInt(f[3], 10, 64) bytes[l[0]] += size total += size } var langs []language for _, l := range languages { if n, ok := bytes[l[0]]; ok { langs = append(langs, language{Name: l[0], Colour: l[1], Bytes: n, Percent: float64(n) * 100 / float64(total)}) delete(bytes, l[0]) } } slices.SortFunc(langs, func(a, b language) int { if a.Bytes != b.Bytes { return int(b.Bytes - a.Bytes) } return strings.Compare(a.Name, b.Name) }) langCache.Lock() langCache.m[repo.ID] = langEntry{head, langs} langCache.Unlock() return langs } type repoHit struct { Repo Language *language score int } type userHit struct { User Profile Profile Repos int Org bool score int } type langCount struct { Name string Colour string Count int Query string } func langQualifier(name string) string { if strings.ContainsRune(name, ' ') { return `language:"` + name + `"` } return "language:" + name } func matchScore(q string, fields ...string) int { if q == "" { return 1 } best := 0 for i, f := range fields { f = strings.ToLower(f) weight := len(fields) - i switch { case f == q: best = max(best, 100*weight) case strings.HasPrefix(f, q): best = max(best, 50*weight) case strings.Contains(f, q): best = max(best, 10*weight) } } return best } func (s *Server) find(w http.ResponseWriter, r *http.Request) { query := strings.TrimSpace(r.URL.Query().Get("q")) kind := r.URL.Query().Get("type") text, quals := parseQuery(query) lang := r.URL.Query().Get("lang") if l, ok := quals["language"]; ok { lang = l } lang = canonicalLanguage(lang) owner, tag := quals["user"], strings.ToLower(quals["tag"]) sort := r.URL.Query().Get("sort") q := strings.ToLower(text) if kind != "users" { kind = "repos" } all, _ := s.store.PublicRepos("", "") var repos []repoHit counts := map[string]*langCount{} for _, repo := range all { langs := s.repoLanguages(repo) var names []string for _, l := range langs { names = append(names, l.Name) } score := matchScore(q, repo.Name, repo.Owner.Name, strings.Join(repo.Tags, " "), strings.Join(names, " "), repo.Description) if score == 0 || (owner != "" && !strings.EqualFold(owner, repo.Owner.Name)) || (tag != "" && !slices.Contains(repo.Tags, tag)) { continue } hit := repoHit{Repo: repo, score: score} if len(langs) > 0 { hit.Language = &langs[0] c, ok := counts[langs[0].Name] if !ok { c = &langCount{Name: langs[0].Name, Colour: langs[0].Colour} counts[langs[0].Name] = c } c.Count++ } if lang != "" && (hit.Language == nil || hit.Language.Name != lang) { continue } hit.Updated = repoUpdated(s.repoDir(repo)) repos = append(repos, hit) } switch sort { case "updated": slices.SortStableFunc(repos, func(a, b repoHit) int { return b.Updated.Compare(a.Updated) }) case "stars": slices.SortStableFunc(repos, func(a, b repoHit) int { return b.Stars - a.Stars }) default: sort = "" slices.SortStableFunc(repos, func(a, b repoHit) int { if a.score != b.score { return b.score - a.score } return b.Updated.Compare(a.Updated) }) } var facets []langCount base := strings.TrimSpace(stripLanguage(query)) for _, c := range counts { c.Query = strings.TrimSpace(base + " " + langQualifier(c.Name)) facets = append(facets, *c) } box := base if lang != "" { box = strings.TrimSpace(base + " " + langQualifier(lang)) } slices.SortFunc(facets, func(a, b langCount) int { if a.Count != b.Count { return b.Count - a.Count } return strings.Compare(a.Name, b.Name) }) var users []userHit for _, u := range s.store.SearchableUsers() { p := s.store.Profile(u.ID) score := matchScore(q, u.Name, p.Display, p.Location, p.Workplace, p.Bio) if score == 0 || (owner != "" && !strings.EqualFold(owner, u.Name)) { continue } owned, _ := s.store.ReposByOwner(u.ID, false) hit := userHit{User: u, Profile: p, Org: s.store.IsOrg(u.ID), score: score} for _, repo := range owned { if repo.Unlisted { continue } hit.Repos++ } if lang != "" { found := false for _, repo := range repos { if repo.Owner.ID == u.ID { found = true break } } if !found { continue } } users = append(users, hit) } slices.SortStableFunc(users, func(a, b userHit) int { if a.score != b.score { return b.score - a.score } return b.Repos - a.Repos }) if len(repos) > 100 { repos = repos[:100] } if len(users) > 100 { users = users[:100] } title := "Search" if query != "" { title = query + " ยท search" } s.render(w, r, "search", page{Title: title, User: s.current(r), Data: map[string]any{ "Query": box, "Base": base, "Type": kind, "Lang": lang, "Sort": sort, "Repos": repos, "Users": users, "Languages": facets, "Facets": true, }}) } var qualifiers = map[string]string{"language": "language", "lang": "language", "user": "user", "owner": "user", "tag": "tag", "topic": "tag"} func queryTokens(query string) []string { var tokens []string var cur strings.Builder quoted := false for _, r := range query { switch { case r == '"': quoted = !quoted case r == ' ' && !quoted: if cur.Len() > 0 { tokens = append(tokens, cur.String()) cur.Reset() } default: cur.WriteRune(r) } } if cur.Len() > 0 { tokens = append(tokens, cur.String()) } return tokens } func parseQuery(query string) (string, map[string]string) { quals := map[string]string{} var words []string for _, tok := range queryTokens(query) { key, value, ok := strings.Cut(tok, ":") if name, known := qualifiers[strings.ToLower(key)]; ok && known && value != "" { quals[name] = strings.TrimPrefix(value, "~") continue } words = append(words, tok) } return strings.Join(words, " "), quals } func stripLanguage(query string) string { var keep []string for _, tok := range queryTokens(query) { key, _, ok := strings.Cut(tok, ":") if ok && qualifiers[strings.ToLower(key)] == "language" { continue } if strings.ContainsRune(tok, ' ') { k, v, _ := strings.Cut(tok, ":") tok = k + `:"` + v + `"` } keep = append(keep, tok) } return strings.Join(keep, " ") } func canonicalLanguage(name string) string { if name == "" { return "" } for _, l := range languages { if strings.EqualFold(l[0], name) { return l[0] } } return name }