diff --git a/docs/configuration.md b/docs/configuration.md index c8bec33..ee5b398 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -43,7 +43,7 @@ that is set but empty counts as not set. | `pdf.remote_images` | `ARCHSIGHT_PDF_REMOTE_IMAGES` | `false` stops the PDF export from fetching images that a page embeds by URL (default: fetched, public hosts only) | | `pdf.max_analyses` | `ARCHSIGHT_PDF_MAX_ANALYSES` | How many `![[Analysis/...]]` embeds one PDF may run (each can take as long as its own timeout). Default `5`; further ones are shown as a note | | `slides.script_hosts` | `ARCHSIGHT_SLIDES_SCRIPT_HOSTS` | `https://` hosts (space or comma separated) from which an HTML [slide deck](/doc/slide-decks) may load scripts, styles and fonts. Default `https://unpkg.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com` | -| `import.cache_root` | `ARCHSIGHT_IMPORT_CACHE_ROOT` | Where the GitHub, GitLab and repository imports clone repositories, and what importers of the resources repository get as `cache_root` / `ARCHSIGHT_CACHE_DIR`. Default `~/.cache/archsight/git`; an import's own `import/config/cacheRoot` wins, see [Importers of the Resources Repository](/doc/import#importers-of-the-resources-repository). Every import makes sure the clones are not indexed by Spotlight (macOS): an empty `.metadata_never_index` in `~/.cache/archsight`, or in the cache root when it is elsewhere | +| `import.cache_root` | `ARCHSIGHT_IMPORT_CACHE_ROOT` | Where the GitHub, GitLab, Gitea and repository imports clone repositories, and what importers of the resources repository get as `cache_root` / `ARCHSIGHT_CACHE_DIR`. Default `~/.cache/archsight/git`; an import's own `import/config/cacheRoot` wins, see [Importers of the Resources Repository](/doc/import#importers-of-the-resources-repository). Every import makes sure the clones are not indexed by Spotlight (macOS): an empty `.metadata_never_index` in `~/.cache/archsight`, or in the cache root when it is elsewhere | ## Command line diff --git a/docs/import.md b/docs/import.md index 9e6f0bb..899b5a5 100644 --- a/docs/import.md +++ b/docs/import.md @@ -1,6 +1,6 @@ # Import System -The import system allows you to declaratively define data imports that generate architecture resources from external sources like GitLab, GitHub, and git repositories. +The import system allows you to declaratively define data imports that generate architecture resources from external sources like GitLab, GitHub, Gitea, and git repositories. ## Overview @@ -159,6 +159,7 @@ Lists repositories from a GitLab instance and generates child Import resources. - `fallbackTeam` - Default team when no contributor match found (propagated to child imports) - `botTeam` - Team for bot-only repositories (propagated to child imports) - `corporateAffixes` - Comma-separated corporate username affixes for team matching, e.g., "ionos,1and1" (propagated to child imports) +- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitlab` for gitlab, `Git:Github` for github; propagated to child imports) **Environment:** - `GITLAB_TOKEN` - Personal access token (required) @@ -175,6 +176,7 @@ Lists repositories from a GitHub organization and generates child Import resourc - `fallbackTeam` - Default team when no contributor match found (propagated to child imports) - `botTeam` - Team for bot-only repositories (propagated to child imports) - `corporateAffixes` - Comma-separated corporate username affixes for team matching, e.g., "ionos,1and1" (propagated to child imports) +- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitlab` for gitlab, `Git:Github` for github; propagated to child imports) **Environment:** - `GITHUB_TOKEN` - GitHub Personal Access Token (required) @@ -184,6 +186,28 @@ Lists repositories from a GitHub organization and generates child Import resourc **Output:** Generates `Import:Repo:github:*` resources for each repository. +### gitea + +Lists repositories from a Gitea (or Forgejo) server and generates child Import resources. + +**Configuration:** +- `host` - Gitea host, e.g. `gitea.example.com` (required) +- `org` - Organization or user to list; without it every repository the token can see is listed +- `scheme` - `https` (default) or `http` +- `verifySSL` - `"false"` skips certificate verification (default: true) +- `rateLimitMs` - Pause between API requests in milliseconds (default: 0) +- `repoOutputPath` - Output path for repository handler results +- `fallbackTeam`, `botTeam`, `corporateAffixes` - Team matching, propagated to child imports like for GitHub +- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitea`; propagated to child imports) +- `defaultVisibility` - Visibility of private repositories (default: `internal`; public ones are `open-source`) + +**Environment:** +- `GITEA_TOKEN` - Access token (required), created at `/user/settings/applications`; scope `read:repository`, plus `read:organization` when `org` is set + +Repositories are cloned over SSH, so the SSH keys of the user must work for the host. + +**Output:** Generates `Import:Repo:gitea::` resources for each repository. + ### repository Analyzes a single git repository and generates a TechnologyArtifact resource. @@ -196,6 +220,7 @@ Analyzes a single git repository and generates a TechnologyArtifact resource. - `sccPath` - Path to scc binary (default: scc) - `fallbackTeam` - Default team when no contributor match found - `botTeam` - Team for bot-only repositories +- `gitService` - TechnologyService the repository is `suppliedBy`, set by the listing import; without it the service is guessed from `gitUrl` (`Git:Github` when it contains `github`, else `Git:Gitlab`) - `corporateAffixes` - Comma-separated corporate username affixes for team matching (e.g., "ionos,1and1"). Enables matching git usernames like `jsmith-ionos` or `ionos-jsmith` to team member "John Smith". Without this, corporate username pattern matching is inactive. **Output:** Generates one TechnologyArtifact resource with: @@ -555,7 +580,7 @@ spec: {} ### Shared state and outputs -- `cache_root` (Ruby) and `ARCHSIGHT_CACHE_DIR` (command) are the directory the GitHub, GitLab and repository +- `cache_root` (Ruby) and `ARCHSIGHT_CACHE_DIR` (command) are the directory the GitHub, GitLab, Gitea and repository imports clone into: `import/config/cacheRoot`, else `import.cache_root` in the [configuration](configuration.md) (`ARCHSIGHT_IMPORT_CACHE_ROOT`), else `~/.cache/archsight/git`. - `database` holds the model as loaded after the dependencies ran (the import command loads the kinds the importers diff --git a/examples/archsight/pages/importing-data.md b/examples/archsight/pages/importing-data.md index e87d411..2a817e8 100644 --- a/examples/archsight/pages/importing-data.md +++ b/examples/archsight/pages/importing-data.md @@ -19,7 +19,7 @@ to fetch them and Archsight writes the resulting YAML into the resources directo layer "flow" { label "Import flow" component "import" { label "**Import**\nresource" } - component "handler" { label "**Handler**\n(GitHub, GitLab, git, REST)" } + component "handler" { label "**Handler**\n(GitHub, GitLab, Gitea, git, REST)" } component "yaml" { label "**Generated**\nYAML" } component "db" { label "Database"; resource "Archsight:Core:Database" } } diff --git a/frontend/src/composables/useFormatting.js b/frontend/src/composables/useFormatting.js index efb5d08..be7cc3d 100644 --- a/frontend/src/composables/useFormatting.js +++ b/frontend/src/composables/useFormatting.js @@ -54,7 +54,7 @@ export function timeAgo(timestamp) { export function iconForUrl(url) { if (/docs\.google\.com\/(document|spreadsheets|presentation)/.test(url)) return 'iconoir-google-docs' if (/github\.com/.test(url)) return 'iconoir-github' - if (/gitlab/.test(url)) return 'iconoir-git-fork' + if (/gitlab|gitea|forgejo|codeberg/.test(url)) return 'iconoir-git-fork' if (/confluence\.|atlassian\.net/.test(url)) return 'iconoir-page-edit' if (/jira\.|atlassian\.net.*jira/.test(url)) return 'iconoir-list' if (/grafana/.test(url)) return 'iconoir-graph-up' @@ -66,7 +66,7 @@ export function iconForUrl(url) { export function categoryForUrl(url) { if (/docs\.google\.com/.test(url)) return 'Documentation' - if (/github\.com|gitlab/.test(url)) return 'Code Repository' + if (/github\.com|gitlab|gitea|forgejo|codeberg/.test(url)) return 'Code Repository' if (/confluence\.|atlassian\.net/.test(url)) return 'Documentation' if (/jira\.|atlassian\.net.*jira/.test(url)) return 'Project Management' if (/grafana|prometheus/.test(url)) return 'Monitoring' diff --git a/frontend/src/composables/useLinks.js b/frontend/src/composables/useLinks.js index e36ef5b..4eca282 100644 --- a/frontend/src/composables/useLinks.js +++ b/frontend/src/composables/useLinks.js @@ -6,6 +6,8 @@ const SYSTEMS = { jira: 'Jira', github: 'GitHub', gitlab: 'GitLab', + gitea: 'Gitea', + forgejo: 'Forgejo', servicenow: 'ServiceNow', grafana: 'Grafana', prometheus: 'Prometheus', diff --git a/lib/archsight/git_source.rb b/lib/archsight/git_source.rb index 446b35c..4fe6bcb 100644 --- a/lib/archsight/git_source.rb +++ b/lib/archsight/git_source.rb @@ -11,7 +11,7 @@ module Archsight # file, its `origin` remote, the checked out branch), nothing goes over the network. # # The web URL is `https:////` of the remote, without the user, password or token a remote URL - # may carry. GitHub files are `/blob//`, GitLab files `/-/blob//`; on other hosts, and for a + # may carry. GitHub files are `/blob//`, GitLab files `/-/blob//`, Gitea/Forgejo files `/src/branch//` (`/src/commit//`); on other hosts, and for a # file git does not track, only the repository is linked. The ref is the checked out branch, the commit when the # HEAD is detached (CI). module GitSource @@ -58,8 +58,10 @@ def host_and_repository(remote) end def file_url(host, repository_url, ref, path) - blob = if host.match?(/(\A|\.)github\./) then "blob" - elsif host.match?(/(\A|\.)gitlab\./) then "-/blob" + blob = case host + when /(\A|\.)github\./ then "blob" + when /(\A|\.)gitlab\./ then "-/blob" + when /(\A|\.)(gitea|forgejo|codeberg)\./ then ref&.match?(/\A\h{40,64}\z/) ? "src/commit" : "src/branch" end return nil unless blob && ref diff --git a/lib/archsight/helpers.rb b/lib/archsight/helpers.rb index c64cdcd..55f4b55 100644 --- a/lib/archsight/helpers.rb +++ b/lib/archsight/helpers.rb @@ -83,6 +83,8 @@ def icon_for_url(url) "iconoir-github" when /gitlab/ "iconoir-git-fork" + when /gitea|forgejo|codeberg/ + "iconoir-git-fork" when /confluence\.|atlassian\.net/ "iconoir-page-edit" when /jira\.|atlassian\.net.*jira/ @@ -138,7 +140,7 @@ def category_for_url(url) case url when %r{docs\.google\.com/(document|spreadsheets|presentation)} "Documentation" - when /github\.com|gitlab/ + when /github\.com|gitlab|gitea|forgejo|codeberg/ "Code Repository" when /confluence\.|atlassian\.net/ "Documentation" diff --git a/lib/archsight/import/handlers/gitea.rb b/lib/archsight/import/handlers/gitea.rb new file mode 100644 index 0000000..9e26262 --- /dev/null +++ b/lib/archsight/import/handlers/gitea.rb @@ -0,0 +1,195 @@ +# frozen_string_literal: true + +require "fileutils" +require "net/http" +require "json" +require "uri" +require "openssl" +require_relative "../handler" +require_relative "../registry" +require_relative "cache_pruner" + +# Gitea handler - lists repositories of a Gitea (or Forgejo) server and generates child Import resources +# +# Configuration: +# import/config/host - Gitea host, e.g. "gitea.example.com" (required) +# import/config/org - Organization (or user) to list; without it every repository the token can see is listed +# import/config/scheme - "https" (default) or "http" +# import/config/verifySSL - "false" to skip certificate verification (default: true) +# import/config/rateLimitMs - Pause between API requests in milliseconds (default: 0) +# import/config/repoOutputPath - Output path for repository handler results (e.g., "generated/repositories.yaml") +# import/config/childCacheTime - Cache time for generated child imports (e.g., "1h", "30m") +# import/config/fallbackTeam - Default team when no contributor match found (propagated to child imports) +# import/config/botTeam - Team for bot-only repositories (propagated to child imports) +# import/config/corporateAffixes - Comma-separated corporate username affixes for team matching (propagated to child imports) +# import/config/defaultVisibility - Default visibility for non-public repos (default: "internal") +# import/config/cacheRoot - Override for the local git cache root (default: ~/.cache/archsight/git) +# +# Environment: +# GITEA_TOKEN - Gitea access token (required) +# Create at: /user/settings/applications (scope read:repository, plus read:organization for `org`) +# +# Output: +# Generates Import:Repo:* resources for each repository +# The repository handler will clone/sync the actual git repositories (over SSH, so the SSH keys must work) +# +# Before generating imports, prunes any cache subdirectory under +# cacheRoot/gitea// that no longer corresponds to a repo returned by +# the API (e.g. a repo that was renamed or deleted), so stale clones don't +# linger and get picked up by later importers. +class Archsight::Import::Handlers::Gitea < Archsight::Import::Handler + # generates child imports (see Handler.generates_imports?) + def self.generates_imports? = true + + include Archsight::Import::Handlers::CachePruner + + # Gitea caps the page size (50 by default), so the listing runs until an empty page + PER_PAGE = 50 + + def execute + @host = config("host") + raise "Missing required config: host" unless @host + + @token = ENV.fetch("GITEA_TOKEN", nil) + raise "Missing required environment variable: GITEA_TOKEN" unless @token + + @org = config("org") + @scheme = config("scheme", default: "https") + @verify_ssl = config("verifySSL", default: "true").to_s != "false" + @rate_limit_ms = config("rateLimitMs", default: "0").to_i + @repo_output_path = config("repoOutputPath") + @child_cache_time = config("childCacheTime") + @default_visibility = config("defaultVisibility", default: "internal") + @host_dir = File.join(Archsight::Import.cache_root(config("cacheRoot")), "gitea", @host) + + progress.update("Fetching repositories from #{@org || @host}") + repos = fetch_all_repos + + if repos.empty? + progress.warn("No repositories found in #{@org || @host}") + return + end + + prune_stale_cache(repos) + + progress.update("Generating #{repos.size} import resources") + generate_repository_imports(repos) + + write_generates_meta + end + + private + + def fetch_all_repos + repos = {} + page = 1 + + loop do + progress.update("Fetching repositories (page #{page})") + batch = fetch_repos_page(page) + break if batch.empty? + + before = repos.size + batch.each { |repo| repos[repo["id"] || repo["fullName"]] = repo } + break if repos.size == before # a server that ignores `page` would repeat the first one forever + + progress.update("Fetched #{repos.size} repositories") + page += 1 + sleep(@rate_limit_ms / 1000.0) if @rate_limit_ms.positive? + end + + repos.values + end + + def fetch_repos_page(page) + data = if @org + make_request("/orgs/#{URI.encode_www_form_component(@org)}/repos", page: page, limit: PER_PAGE) + else + # the search wraps the repositories: { ok: true, data: [...] } + make_request("/repos/search", page: page, limit: PER_PAGE)["data"] + end + + Array(data).map do |repo| + { + "id" => repo["id"], + "name" => repo["name"], + "fullName" => repo["full_name"], + "owner" => repo.dig("owner", "login") || repo["full_name"].to_s.split("/").first, + "isArchived" => repo["archived"], + "private" => repo["private"], + "sshUrl" => repo["ssh_url"], + "url" => repo["clone_url"] || repo["html_url"] + } + end + end + + def make_request(path, params = {}) + uri = URI("#{@scheme}://#{@host}/api/v1#{path}") + uri.query = URI.encode_www_form(params) unless params.empty? + + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = uri.scheme == "https" + http.verify_mode = OpenSSL::SSL::VERIFY_NONE if http.use_ssl? && !@verify_ssl + + request = Net::HTTP::Get.new(uri) + request["Authorization"] = "token #{@token}" + request["Accept"] = "application/json" + + handle_response(http.request(request)) + end + + def handle_response(response) + case response + when Net::HTTPSuccess + JSON.parse(response.body) + when Net::HTTPUnauthorized + raise "Gitea API error: 401 Unauthorized - Invalid or expired GITEA_TOKEN" + when Net::HTTPForbidden + raise "Gitea API error: 403 Forbidden - Check token permissions" + when Net::HTTPNotFound + raise "Gitea API error: 404 Not Found - #{@org ? "Organization '#{@org}'" : "API"} not found or not accessible on #{@host}" + else + raise "Gitea API error: #{response.code} #{response.message}" + end + end + + # Each owner has its own directory; owners that are not listed are left alone + def prune_stale_cache(repos) + repos.group_by { |repo| repo["owner"] }.each do |owner, owned| + prune_stale_cache_entries(File.join(@host_dir, owner), owned.map { |repo| repo["name"] }) + end + end + + def generate_repository_imports(repos) + yaml_documents = repos.map do |repo| + child_annotations = {} + child_annotations["import/outputPath"] = @repo_output_path if @repo_output_path + child_annotations["import/cacheTime"] = @child_cache_time if @child_cache_time + + child_config = { + "path" => File.join(@host_dir, repo["owner"], repo["name"]), + "gitUrl" => repo["sshUrl"] || repo["url"], + "archived" => repo["isArchived"].to_s, + "visibility" => repo["private"] == false ? "open-source" : @default_visibility + } + child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam") + child_config["botTeam"] = config("botTeam") if config("botTeam") + child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes") + child_config["gitService"] = config("gitService", default: "Git:Gitea") + + import_yaml( + name: "Import:Repo:gitea:#{repo["owner"]}:#{repo["name"]}", + handler: "repository", + config: child_config, + annotations: child_annotations + ) + end + + # Add self-marker with generated/at for caching + yaml_documents << self_marker + + write_yaml(yaml_documents.map { |doc| YAML.dump(doc) }.join("\n")) + end +end + +Archsight::Import::Registry.register("gitea", Archsight::Import::Handlers::Gitea) diff --git a/lib/archsight/import/handlers/github.rb b/lib/archsight/import/handlers/github.rb index 148587b..4a2db3b 100644 --- a/lib/archsight/import/handlers/github.rb +++ b/lib/archsight/import/handlers/github.rb @@ -165,6 +165,7 @@ def generate_repository_imports(repos) child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam") child_config["botTeam"] = config("botTeam") if config("botTeam") child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes") + child_config["gitService"] = config("gitService", default: "Git:Github") import_yaml( name: "Import:Repo:github:#{@org}:#{repo_name}", diff --git a/lib/archsight/import/handlers/gitlab.rb b/lib/archsight/import/handlers/gitlab.rb index a87a0b6..6dfc541 100644 --- a/lib/archsight/import/handlers/gitlab.rb +++ b/lib/archsight/import/handlers/gitlab.rb @@ -214,6 +214,7 @@ def generate_repository_imports(projects) child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam") child_config["botTeam"] = config("botTeam") if config("botTeam") child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes") + child_config["gitService"] = config("gitService", default: "Git:Gitlab") import_yaml( name: "Import:Repo:gitlab:#{dir_name}", diff --git a/lib/archsight/import/handlers/repository.rb b/lib/archsight/import/handlers/repository.rb index 24185ec..8728f48 100644 --- a/lib/archsight/import/handlers/repository.rb +++ b/lib/archsight/import/handlers/repository.rb @@ -356,10 +356,7 @@ def build_technology_artifact(path, scc_data, git_data, team_result = nil, licen spec = {} # Technology component (Git provider) - if git_url - provider = git_url.include?("github") ? "Git:Github" : "Git:Gitlab" - spec["suppliedBy"] = { "technologyComponents" => [provider] } - end + spec["suppliedBy"] = { "technologyComponents" => [git_service(git_url)] } if git_url # Team relations from contributor matching if team_result @@ -386,6 +383,12 @@ def extract_git_url(path) url_line.split("=").last.strip end + # The TechnologyService that supplies the repository: `gitService` as passed by the listing import + # (github, gitlab, gitea), else guessed from the clone URL for repository imports written by hand + def git_service(git_url) + config("gitService") || (git_url.include?("github") ? "Git:Github" : "Git:Gitlab") + end + def repository_name(git_url_or_path) if git_url_or_path.include?(":") # Git URL format diff --git a/test/git_source_test.rb b/test/git_source_test.rb index f1f258b..dbdd161 100644 --- a/test/git_source_test.rb +++ b/test/git_source_test.rb @@ -43,6 +43,20 @@ def test_a_gitlab_file_links_to_its_blob assert_equal "https://gitlab.example.com/group/sub/architecture/-/blob/main/arch/pages/my%20page.md", G.locate(@file).url end + def test_a_gitea_file_links_to_its_branch + git("remote", "add", "origin", "git@gitea.example.com:group/architecture.git") + + assert_equal "https://gitea.example.com/group/architecture/src/branch/main/arch/pages/my%20page.md", G.locate(@file).url + end + + def test_a_gitea_detached_head_links_to_the_commit + git("remote", "add", "origin", "git@gitea.example.com:group/architecture.git") + git("checkout", "-q", "--detach") + sha, = Open3.capture2("git", "-C", @dir, "rev-parse", "HEAD") + + assert_equal "https://gitea.example.com/group/architecture/src/commit/#{sha.strip}/arch/pages/my%20page.md", G.locate(@file).url + end + def test_a_detached_head_links_to_the_commit git("remote", "add", "origin", "https://github.com/example/architecture") git("checkout", "-q", "--detach") diff --git a/test/helpers_test.rb b/test/helpers_test.rb index 4f3c556..52af333 100644 --- a/test/helpers_test.rb +++ b/test/helpers_test.rb @@ -66,6 +66,11 @@ def test_icon_for_gitlab assert_equal "iconoir-git-fork", icon_for_url("https://gitlab.com/owner/repo") end + def test_icon_and_category_for_gitea + assert_equal "iconoir-git-fork", icon_for_url("https://gitea.example.com/owner/repo") + assert_equal "Code Repository", category_for_url("https://gitea.example.com/owner/repo") + end + def test_icon_for_google_docs assert_equal "iconoir-google-docs", icon_for_url("https://docs.google.com/document/d/123") end diff --git a/test/import/handlers/gitea_handler_test.rb b/test/import/handlers/gitea_handler_test.rb new file mode 100644 index 0000000..5a740bb --- /dev/null +++ b/test/import/handlers/gitea_handler_test.rb @@ -0,0 +1,210 @@ +# frozen_string_literal: true + +require "test_helper" +require "tmpdir" +require "stringio" +require "webmock/minitest" +require "archsight/import/handlers/gitea" +require "archsight/import/progress" + +class GiteaHandlerTest < Minitest::Test + HOST = "gitea.example.com" + API = "https://#{HOST}/api/v1".freeze + + def setup + @resources_dir = Dir.mktmpdir + @original_token = ENV.fetch("GITEA_TOKEN", nil) + ENV["GITEA_TOKEN"] = "test-token" + end + + def teardown + FileUtils.rm_rf(@resources_dir) + if @original_token + ENV["GITEA_TOKEN"] = @original_token + else + ENV.delete("GITEA_TOKEN") + end + end + + def repo(name, owner: "test-org", private: false, archived: false, id: nil) + { + "id" => id || name.hash.abs, "name" => name, "full_name" => "#{owner}/#{name}", "owner" => { "login" => owner }, + "archived" => archived, "private" => private, + "ssh_url" => "git@#{HOST}:#{owner}/#{name}.git", "clone_url" => "https://#{HOST}/#{owner}/#{name}.git", + "html_url" => "https://#{HOST}/#{owner}/#{name}" + } + end + + def stub_org(repos, org: "test-org") + stub_request(:get, "#{API}/orgs/#{org}/repos?limit=50&page=1").to_return(status: 200, body: repos.to_json) + stub_request(:get, "#{API}/orgs/#{org}/repos?limit=50&page=2").to_return(status: 200, body: [].to_json) + end + + def output + File.read(File.join(@resources_dir, "generated", "Import_Gitea.yaml")) + end + + def test_missing_token_raises_error + ENV.delete("GITEA_TOKEN") + error = assert_raises(RuntimeError) { create_handler(org: "test-org").execute } + + assert_equal "Missing required environment variable: GITEA_TOKEN", error.message + end + + def test_missing_host_raises_error + error = assert_raises(RuntimeError) { create_handler(host: nil, org: "test-org").execute } + + assert_equal "Missing required config: host", error.message + end + + def test_lists_an_organization_with_token_header_and_generates_child_imports + stub_org([repo("repo1"), repo("repo2", private: true, archived: true)]) + stub_request(:get, "#{API}/orgs/test-org/repos?limit=50&page=1").with(headers: { "Authorization" => "token test-token" }) + .to_return(status: 200, body: [repo("repo1"), repo("repo2", private: true, archived: true)].to_json) + create_handler(org: "test-org").execute + + assert_includes output, "Import:Repo:gitea:test-org:repo1" + assert_includes output, "git@#{HOST}:test-org/repo1.git" + assert_includes output, "open-source" + assert_includes output, "Import:Repo:gitea:test-org:repo2" + assert_match(/archived: 'true'|archived: "true"|archived: true/, output) + assert_includes output, "internal" + end + + def test_paginates_until_an_empty_page + first = (1..50).map { |i| repo("r#{i}", id: i) } + stub_request(:get, "#{API}/orgs/test-org/repos?limit=50&page=1").to_return(status: 200, body: first.to_json) + stub_request(:get, "#{API}/orgs/test-org/repos?limit=50&page=2").to_return(status: 200, body: [repo("last", id: 99)].to_json) + stub_request(:get, "#{API}/orgs/test-org/repos?limit=50&page=3").to_return(status: 200, body: [].to_json) + create_handler(org: "test-org").execute + + assert_includes output, "Import:Repo:gitea:test-org:r50" + assert_includes output, "Import:Repo:gitea:test-org:last" + end + + def test_a_server_that_repeats_the_first_page_does_not_loop_forever + stub_request(:get, %r{#{API}/orgs/test-org/repos}).to_return(status: 200, body: [repo("repo1", id: 1)].to_json) + create_handler(org: "test-org").execute + + assert_includes output, "Import:Repo:gitea:test-org:repo1" + end + + def test_without_org_searches_all_visible_repositories + body = { "ok" => true, "data" => [repo("a", owner: "one", id: 1), repo("b", owner: "two", id: 2)] }.to_json + stub_request(:get, "#{API}/repos/search?limit=50&page=1").to_return(status: 200, body: body) + stub_request(:get, "#{API}/repos/search?limit=50&page=2").to_return(status: 200, body: { "ok" => true, "data" => [] }.to_json) + create_handler(org: nil).execute + + assert_includes output, "Import:Repo:gitea:one:a" + assert_includes output, "Import:Repo:gitea:two:b" + end + + def test_http_error_handling + { 401 => "401 Unauthorized", 403 => "403 Forbidden", 404 => "404 Not Found", 500 => "500" }.each do |code, text| + stub_request(:get, "#{API}/orgs/test-org/repos?limit=50&page=1").to_return(status: code, body: "") + error = assert_raises(RuntimeError) { create_handler(org: "test-org").execute } + + assert_includes error.message, "Gitea API error: #{text}" + end + end + + def test_propagates_team_config_to_child_imports + stub_org([repo("repo1")]) + create_handler(org: "test-org", fallback_team: "Team:Platform", bot_team: "Team:Bots", corporate_affixes: "acme").execute + + assert_includes output, "Team:Platform" + assert_includes output, "Team:Bots" + assert_includes output, "acme" + end + + def test_passes_gitea_service_to_child_imports + stub_org([repo("repo1")]) + create_handler(org: "test-org").execute + + assert_includes output, "Git:Gitea" + end + + def test_omits_team_config_when_not_set + stub_org([repo("repo1")]) + create_handler(org: "test-org").execute + + refute_includes output, "fallbackTeam" + refute_includes output, "botTeam" + end + + def test_prunes_stale_cache_directory_for_renamed_repo + cache_root = Dir.mktmpdir + owner_dir = File.join(cache_root, "gitea", HOST, "test-org") + FileUtils.mkdir_p(File.join(owner_dir, "stale-repo")) + FileUtils.mkdir_p(File.join(owner_dir, "repo1")) + old = Time.now - 7200 + File.utime(old, old, File.join(owner_dir, "stale-repo")) + stub_org([repo("repo1")]) + create_handler(org: "test-org", cache_root: cache_root).execute + + refute_path_exists File.join(owner_dir, "stale-repo") + assert_path_exists File.join(owner_dir, "repo1") + ensure + FileUtils.rm_rf(cache_root) + end + + def test_pruning_leaves_owners_that_are_not_listed_alone + cache_root = Dir.mktmpdir + other = File.join(cache_root, "gitea", HOST, "other-owner", "old-repo") + FileUtils.mkdir_p(other) + old = Time.now - 7200 + File.utime(old, old, other) + stub_org([repo("repo1")]) + create_handler(org: "test-org", cache_root: cache_root).execute + + assert_path_exists other + ensure + FileUtils.rm_rf(cache_root) + end + + def test_the_clone_path_includes_host_and_owner + cache_root = Dir.mktmpdir + stub_org([repo("repo1")]) + create_handler(org: "test-org", cache_root: cache_root).execute + + assert_includes output, File.join(cache_root, "gitea", HOST, "test-org", "repo1") + ensure + FileUtils.rm_rf(cache_root) + end + + def test_registered_as_gitea + Archsight::Import::Registry.restore_builtin! # other tests clear the registry + + assert_equal Archsight::Import::Handlers::Gitea, Archsight::Import::Registry["gitea"] + end + + private + + def create_handler(org:, host: HOST, fallback_team: nil, bot_team: nil, corporate_affixes: nil, cache_root: nil) + annotations = { "import/handler" => "gitea", "import/config/repoOutputPath" => "generated/repos.yaml" } + annotations["import/config/host"] = host if host + annotations["import/config/org"] = org if org + annotations["import/config/fallbackTeam"] = fallback_team if fallback_team + annotations["import/config/botTeam"] = bot_team if bot_team + annotations["import/config/corporateAffixes"] = corporate_affixes if corporate_affixes + annotations["import/config/cacheRoot"] = cache_root if cache_root + + raw = { "apiVersion" => "architecture/v1alpha1", "kind" => "Import", + "metadata" => { "name" => "Import:Gitea", "annotations" => annotations }, "spec" => {} } + Archsight::Import::Handlers::Gitea.new(MockGiteaImport.new(raw), database: nil, resources_dir: @resources_dir, + progress: Archsight::Import::Progress.new(output: StringIO.new)) + end + + class MockGiteaImport + attr_reader :raw, :name, :annotations, :path_ref + + PathRef = Struct.new(:path) + + def initialize(raw) + @raw = raw + @name = raw.dig("metadata", "name") + @annotations = raw.dig("metadata", "annotations") || {} + @path_ref = PathRef.new("/tmp/gitea-test.yaml") + end + end +end diff --git a/test/import/handlers/repository_test.rb b/test/import/handlers/repository_test.rb index 311e607..090fed9 100644 --- a/test/import/handlers/repository_test.rb +++ b/test/import/handlers/repository_test.rb @@ -53,6 +53,18 @@ def test_born_clone_is_fast_forwarded_as_before assert_equal "second", File.read(File.join(@clone, "README")).strip end + def test_git_service_comes_from_the_listing_import + h = handler + h.import_resource.annotations["import/config/gitService"] = "Git:Gitea" + + assert_equal "Git:Gitea", h.send(:git_service, "gitea@git.example.com:org/repo.git") + end + + def test_git_service_falls_back_to_the_clone_url + assert_equal "Git:Github", handler.send(:git_service, "git@github.com:org/repo.git") + assert_equal "Git:Gitlab", handler.send(:git_service, "git@gitlab.example.com:org/repo.git") + end + private def push_commit(text = "hello")