Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ that is set but empty counts as not set.
| `pdf.remote_images` | `ARCHSIGHT_PDF_REMOTE_IMAGES` | `false` stops the PDF export from fetching images that a page embeds by URL (default: fetched, public hosts only) |
| `pdf.max_analyses` | `ARCHSIGHT_PDF_MAX_ANALYSES` | How many `![[Analysis/...]]` embeds one PDF may run (each can take as long as its own timeout). Default `5`; further ones are shown as a note |
| `slides.script_hosts` | `ARCHSIGHT_SLIDES_SCRIPT_HOSTS` | `https://` hosts (space or comma separated) from which an HTML [slide deck](/doc/slide-decks) may load scripts, styles and fonts. Default `https://unpkg.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com` |
| `import.cache_root` | `ARCHSIGHT_IMPORT_CACHE_ROOT` | Where the GitHub, GitLab and repository imports clone repositories, and what importers of the resources repository get as `cache_root` / `ARCHSIGHT_CACHE_DIR`. Default `~/.cache/archsight/git`; an import's own `import/config/cacheRoot` wins, see [Importers of the Resources Repository](/doc/import#importers-of-the-resources-repository). Every import makes sure the clones are not indexed by Spotlight (macOS): an empty `.metadata_never_index` in `~/.cache/archsight`, or in the cache root when it is elsewhere |
| `import.cache_root` | `ARCHSIGHT_IMPORT_CACHE_ROOT` | Where the GitHub, GitLab, Gitea and repository imports clone repositories, and what importers of the resources repository get as `cache_root` / `ARCHSIGHT_CACHE_DIR`. Default `~/.cache/archsight/git`; an import's own `import/config/cacheRoot` wins, see [Importers of the Resources Repository](/doc/import#importers-of-the-resources-repository). Every import makes sure the clones are not indexed by Spotlight (macOS): an empty `.metadata_never_index` in `~/.cache/archsight`, or in the cache root when it is elsewhere |

## Command line

Expand Down
29 changes: 27 additions & 2 deletions docs/import.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Import System

The import system allows you to declaratively define data imports that generate architecture resources from external sources like GitLab, GitHub, and git repositories.
The import system allows you to declaratively define data imports that generate architecture resources from external sources like GitLab, GitHub, Gitea, and git repositories.

## Overview

Expand Down Expand Up @@ -159,6 +159,7 @@ Lists repositories from a GitLab instance and generates child Import resources.
- `fallbackTeam` - Default team when no contributor match found (propagated to child imports)
- `botTeam` - Team for bot-only repositories (propagated to child imports)
- `corporateAffixes` - Comma-separated corporate username affixes for team matching, e.g., "ionos,1and1" (propagated to child imports)
- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitlab` for gitlab, `Git:Github` for github; propagated to child imports)

**Environment:**
- `GITLAB_TOKEN` - Personal access token (required)
Expand All @@ -175,6 +176,7 @@ Lists repositories from a GitHub organization and generates child Import resourc
- `fallbackTeam` - Default team when no contributor match found (propagated to child imports)
- `botTeam` - Team for bot-only repositories (propagated to child imports)
- `corporateAffixes` - Comma-separated corporate username affixes for team matching, e.g., "ionos,1and1" (propagated to child imports)
- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitlab` for gitlab, `Git:Github` for github; propagated to child imports)

**Environment:**
- `GITHUB_TOKEN` - GitHub Personal Access Token (required)
Expand All @@ -184,6 +186,28 @@ Lists repositories from a GitHub organization and generates child Import resourc

**Output:** Generates `Import:Repo:github:*` resources for each repository.

### gitea

Lists repositories from a Gitea (or Forgejo) server and generates child Import resources.

**Configuration:**
- `host` - Gitea host, e.g. `gitea.example.com` (required)
- `org` - Organization or user to list; without it every repository the token can see is listed
- `scheme` - `https` (default) or `http`
- `verifySSL` - `"false"` skips certificate verification (default: true)
- `rateLimitMs` - Pause between API requests in milliseconds (default: 0)
- `repoOutputPath` - Output path for repository handler results
- `fallbackTeam`, `botTeam`, `corporateAffixes` - Team matching, propagated to child imports like for GitHub
- `gitService` - TechnologyService that supplies the repositories (default: `Git:Gitea`; propagated to child imports)
- `defaultVisibility` - Visibility of private repositories (default: `internal`; public ones are `open-source`)

**Environment:**
- `GITEA_TOKEN` - Access token (required), created at `<host>/user/settings/applications`; scope `read:repository`, plus `read:organization` when `org` is set

Repositories are cloned over SSH, so the SSH keys of the user must work for the host.

**Output:** Generates `Import:Repo:gitea:<owner>:<repo>` resources for each repository.

### repository

Analyzes a single git repository and generates a TechnologyArtifact resource.
Expand All @@ -196,6 +220,7 @@ Analyzes a single git repository and generates a TechnologyArtifact resource.
- `sccPath` - Path to scc binary (default: scc)
- `fallbackTeam` - Default team when no contributor match found
- `botTeam` - Team for bot-only repositories
- `gitService` - TechnologyService the repository is `suppliedBy`, set by the listing import; without it the service is guessed from `gitUrl` (`Git:Github` when it contains `github`, else `Git:Gitlab`)
- `corporateAffixes` - Comma-separated corporate username affixes for team matching (e.g., "ionos,1and1"). Enables matching git usernames like `jsmith-ionos` or `ionos-jsmith` to team member "John Smith". Without this, corporate username pattern matching is inactive.

**Output:** Generates one TechnologyArtifact resource with:
Expand Down Expand Up @@ -555,7 +580,7 @@ spec: {}

### Shared state and outputs

- `cache_root` (Ruby) and `ARCHSIGHT_CACHE_DIR` (command) are the directory the GitHub, GitLab and repository
- `cache_root` (Ruby) and `ARCHSIGHT_CACHE_DIR` (command) are the directory the GitHub, GitLab, Gitea and repository
imports clone into: `import/config/cacheRoot`, else `import.cache_root` in the
[configuration](configuration.md) (`ARCHSIGHT_IMPORT_CACHE_ROOT`), else `~/.cache/archsight/git`.
- `database` holds the model as loaded after the dependencies ran (the import command loads the kinds the importers
Expand Down
2 changes: 1 addition & 1 deletion examples/archsight/pages/importing-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ to fetch them and Archsight writes the resulting YAML into the resources directo
layer "flow" {
label "Import flow"
component "import" { label "**Import**\nresource" }
component "handler" { label "**Handler**\n(GitHub, GitLab, git, REST)" }
component "handler" { label "**Handler**\n(GitHub, GitLab, Gitea, git, REST)" }
component "yaml" { label "**Generated**\nYAML" }
component "db" { label "Database"; resource "Archsight:Core:Database" }
}
Expand Down
4 changes: 2 additions & 2 deletions frontend/src/composables/useFormatting.js
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ export function timeAgo(timestamp) {
export function iconForUrl(url) {
if (/docs\.google\.com\/(document|spreadsheets|presentation)/.test(url)) return 'iconoir-google-docs'
if (/github\.com/.test(url)) return 'iconoir-github'
if (/gitlab/.test(url)) return 'iconoir-git-fork'
if (/gitlab|gitea|forgejo|codeberg/.test(url)) return 'iconoir-git-fork'
if (/confluence\.|atlassian\.net/.test(url)) return 'iconoir-page-edit'
if (/jira\.|atlassian\.net.*jira/.test(url)) return 'iconoir-list'
if (/grafana/.test(url)) return 'iconoir-graph-up'
Expand All @@ -66,7 +66,7 @@ export function iconForUrl(url) {

export function categoryForUrl(url) {
if (/docs\.google\.com/.test(url)) return 'Documentation'
if (/github\.com|gitlab/.test(url)) return 'Code Repository'
if (/github\.com|gitlab|gitea|forgejo|codeberg/.test(url)) return 'Code Repository'
if (/confluence\.|atlassian\.net/.test(url)) return 'Documentation'
if (/jira\.|atlassian\.net.*jira/.test(url)) return 'Project Management'
if (/grafana|prometheus/.test(url)) return 'Monitoring'
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/composables/useLinks.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ const SYSTEMS = {
jira: 'Jira',
github: 'GitHub',
gitlab: 'GitLab',
gitea: 'Gitea',
forgejo: 'Forgejo',
servicenow: 'ServiceNow',
grafana: 'Grafana',
prometheus: 'Prometheus',
Expand Down
8 changes: 5 additions & 3 deletions lib/archsight/git_source.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ module Archsight
# file, its `origin` remote, the checked out branch), nothing goes over the network.
#
# The web URL is `https://<host>/<owner>/<repo>` of the remote, without the user, password or token a remote URL
# may carry. GitHub files are `/blob/<ref>/<path>`, GitLab files `/-/blob/<ref>/<path>`; on other hosts, and for a
# may carry. GitHub files are `/blob/<ref>/<path>`, GitLab files `/-/blob/<ref>/<path>`, Gitea/Forgejo files `/src/branch/<ref>/<path>` (`/src/commit/<sha>/<path>`); on other hosts, and for a
# file git does not track, only the repository is linked. The ref is the checked out branch, the commit when the
# HEAD is detached (CI).
module GitSource
Expand Down Expand Up @@ -58,8 +58,10 @@ def host_and_repository(remote)
end

def file_url(host, repository_url, ref, path)
blob = if host.match?(/(\A|\.)github\./) then "blob"
elsif host.match?(/(\A|\.)gitlab\./) then "-/blob"
blob = case host
when /(\A|\.)github\./ then "blob"
when /(\A|\.)gitlab\./ then "-/blob"
when /(\A|\.)(gitea|forgejo|codeberg)\./ then ref&.match?(/\A\h{40,64}\z/) ? "src/commit" : "src/branch"
end
return nil unless blob && ref

Expand Down
4 changes: 3 additions & 1 deletion lib/archsight/helpers.rb
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ def icon_for_url(url)
"iconoir-github"
when /gitlab/
"iconoir-git-fork"
when /gitea|forgejo|codeberg/
"iconoir-git-fork"
when /confluence\.|atlassian\.net/
"iconoir-page-edit"
when /jira\.|atlassian\.net.*jira/
Expand Down Expand Up @@ -138,7 +140,7 @@ def category_for_url(url)
case url
when %r{docs\.google\.com/(document|spreadsheets|presentation)}
"Documentation"
when /github\.com|gitlab/
when /github\.com|gitlab|gitea|forgejo|codeberg/
"Code Repository"
when /confluence\.|atlassian\.net/
"Documentation"
Expand Down
195 changes: 195 additions & 0 deletions lib/archsight/import/handlers/gitea.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
# frozen_string_literal: true

require "fileutils"
require "net/http"
require "json"
require "uri"
require "openssl"
require_relative "../handler"
require_relative "../registry"
require_relative "cache_pruner"

# Gitea handler - lists repositories of a Gitea (or Forgejo) server and generates child Import resources
#
# Configuration:
# import/config/host - Gitea host, e.g. "gitea.example.com" (required)
# import/config/org - Organization (or user) to list; without it every repository the token can see is listed
# import/config/scheme - "https" (default) or "http"
# import/config/verifySSL - "false" to skip certificate verification (default: true)
# import/config/rateLimitMs - Pause between API requests in milliseconds (default: 0)
# import/config/repoOutputPath - Output path for repository handler results (e.g., "generated/repositories.yaml")
# import/config/childCacheTime - Cache time for generated child imports (e.g., "1h", "30m")
# import/config/fallbackTeam - Default team when no contributor match found (propagated to child imports)
# import/config/botTeam - Team for bot-only repositories (propagated to child imports)
# import/config/corporateAffixes - Comma-separated corporate username affixes for team matching (propagated to child imports)
# import/config/defaultVisibility - Default visibility for non-public repos (default: "internal")
# import/config/cacheRoot - Override for the local git cache root (default: ~/.cache/archsight/git)
#
# Environment:
# GITEA_TOKEN - Gitea access token (required)
# Create at: <host>/user/settings/applications (scope read:repository, plus read:organization for `org`)
#
# Output:
# Generates Import:Repo:* resources for each repository
# The repository handler will clone/sync the actual git repositories (over SSH, so the SSH keys must work)
#
# Before generating imports, prunes any cache subdirectory under
# cacheRoot/gitea/<host>/<owner> that no longer corresponds to a repo returned by
# the API (e.g. a repo that was renamed or deleted), so stale clones don't
# linger and get picked up by later importers.
class Archsight::Import::Handlers::Gitea < Archsight::Import::Handler
# generates child imports (see Handler.generates_imports?)
def self.generates_imports? = true

include Archsight::Import::Handlers::CachePruner

# Gitea caps the page size (50 by default), so the listing runs until an empty page
PER_PAGE = 50

def execute
@host = config("host")
raise "Missing required config: host" unless @host

@token = ENV.fetch("GITEA_TOKEN", nil)
raise "Missing required environment variable: GITEA_TOKEN" unless @token

@org = config("org")
@scheme = config("scheme", default: "https")
@verify_ssl = config("verifySSL", default: "true").to_s != "false"
@rate_limit_ms = config("rateLimitMs", default: "0").to_i
@repo_output_path = config("repoOutputPath")
@child_cache_time = config("childCacheTime")
@default_visibility = config("defaultVisibility", default: "internal")
@host_dir = File.join(Archsight::Import.cache_root(config("cacheRoot")), "gitea", @host)

progress.update("Fetching repositories from #{@org || @host}")
repos = fetch_all_repos

if repos.empty?
progress.warn("No repositories found in #{@org || @host}")
return
end

prune_stale_cache(repos)

progress.update("Generating #{repos.size} import resources")
generate_repository_imports(repos)

write_generates_meta
end

private

def fetch_all_repos
repos = {}
page = 1

loop do
progress.update("Fetching repositories (page #{page})")
batch = fetch_repos_page(page)
break if batch.empty?

before = repos.size
batch.each { |repo| repos[repo["id"] || repo["fullName"]] = repo }
break if repos.size == before # a server that ignores `page` would repeat the first one forever

progress.update("Fetched #{repos.size} repositories")
page += 1
sleep(@rate_limit_ms / 1000.0) if @rate_limit_ms.positive?
end

repos.values
end

def fetch_repos_page(page)
data = if @org
make_request("/orgs/#{URI.encode_www_form_component(@org)}/repos", page: page, limit: PER_PAGE)
else
# the search wraps the repositories: { ok: true, data: [...] }
make_request("/repos/search", page: page, limit: PER_PAGE)["data"]
end

Array(data).map do |repo|
{
"id" => repo["id"],
"name" => repo["name"],
"fullName" => repo["full_name"],
"owner" => repo.dig("owner", "login") || repo["full_name"].to_s.split("/").first,
"isArchived" => repo["archived"],
"private" => repo["private"],
"sshUrl" => repo["ssh_url"],
"url" => repo["clone_url"] || repo["html_url"]
}
end
end

def make_request(path, params = {})
uri = URI("#{@scheme}://#{@host}/api/v1#{path}")
uri.query = URI.encode_www_form(params) unless params.empty?

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = uri.scheme == "https"
http.verify_mode = OpenSSL::SSL::VERIFY_NONE if http.use_ssl? && !@verify_ssl

request = Net::HTTP::Get.new(uri)
request["Authorization"] = "token #{@token}"
request["Accept"] = "application/json"

handle_response(http.request(request))
end

def handle_response(response)
case response
when Net::HTTPSuccess
JSON.parse(response.body)
when Net::HTTPUnauthorized
raise "Gitea API error: 401 Unauthorized - Invalid or expired GITEA_TOKEN"
when Net::HTTPForbidden
raise "Gitea API error: 403 Forbidden - Check token permissions"
when Net::HTTPNotFound
raise "Gitea API error: 404 Not Found - #{@org ? "Organization '#{@org}'" : "API"} not found or not accessible on #{@host}"
else
raise "Gitea API error: #{response.code} #{response.message}"
end
end

# Each owner has its own directory; owners that are not listed are left alone
def prune_stale_cache(repos)
repos.group_by { |repo| repo["owner"] }.each do |owner, owned|
prune_stale_cache_entries(File.join(@host_dir, owner), owned.map { |repo| repo["name"] })
end
end

def generate_repository_imports(repos)
yaml_documents = repos.map do |repo|
child_annotations = {}
child_annotations["import/outputPath"] = @repo_output_path if @repo_output_path
child_annotations["import/cacheTime"] = @child_cache_time if @child_cache_time

child_config = {
"path" => File.join(@host_dir, repo["owner"], repo["name"]),
"gitUrl" => repo["sshUrl"] || repo["url"],
"archived" => repo["isArchived"].to_s,
"visibility" => repo["private"] == false ? "open-source" : @default_visibility
}
child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam")
child_config["botTeam"] = config("botTeam") if config("botTeam")
child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes")
child_config["gitService"] = config("gitService", default: "Git:Gitea")

import_yaml(
name: "Import:Repo:gitea:#{repo["owner"]}:#{repo["name"]}",
handler: "repository",
config: child_config,
annotations: child_annotations
)
end

# Add self-marker with generated/at for caching
yaml_documents << self_marker

write_yaml(yaml_documents.map { |doc| YAML.dump(doc) }.join("\n"))
end
end

Archsight::Import::Registry.register("gitea", Archsight::Import::Handlers::Gitea)
1 change: 1 addition & 0 deletions lib/archsight/import/handlers/github.rb
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ def generate_repository_imports(repos)
child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam")
child_config["botTeam"] = config("botTeam") if config("botTeam")
child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes")
child_config["gitService"] = config("gitService", default: "Git:Github")

import_yaml(
name: "Import:Repo:github:#{@org}:#{repo_name}",
Expand Down
1 change: 1 addition & 0 deletions lib/archsight/import/handlers/gitlab.rb
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ def generate_repository_imports(projects)
child_config["fallbackTeam"] = config("fallbackTeam") if config("fallbackTeam")
child_config["botTeam"] = config("botTeam") if config("botTeam")
child_config["corporateAffixes"] = config("corporateAffixes") if config("corporateAffixes")
child_config["gitService"] = config("gitService", default: "Git:Gitlab")

import_yaml(
name: "Import:Repo:gitlab:#{dir_name}",
Expand Down
11 changes: 7 additions & 4 deletions lib/archsight/import/handlers/repository.rb
Original file line number Diff line number Diff line change
Expand Up @@ -356,10 +356,7 @@ def build_technology_artifact(path, scc_data, git_data, team_result = nil, licen
spec = {}

# Technology component (Git provider)
if git_url
provider = git_url.include?("github") ? "Git:Github" : "Git:Gitlab"
spec["suppliedBy"] = { "technologyComponents" => [provider] }
end
spec["suppliedBy"] = { "technologyComponents" => [git_service(git_url)] } if git_url

# Team relations from contributor matching
if team_result
Expand All @@ -386,6 +383,12 @@ def extract_git_url(path)
url_line.split("=").last.strip
end

# The TechnologyService that supplies the repository: `gitService` as passed by the listing import
# (github, gitlab, gitea), else guessed from the clone URL for repository imports written by hand
def git_service(git_url)
config("gitService") || (git_url.include?("github") ? "Git:Github" : "Git:Gitlab")
end

def repository_name(git_url_or_path)
if git_url_or_path.include?(":")
# Git URL format
Expand Down
Loading
Loading