Consider creating a trusted list of IPs for fine tuned SSR (to filter out aggressive bots)

[tdonohue]

Is your feature request related to a problem? Please describe.

In order to provide better protections against aggressive bots, DSpace should consider adding a "trusted IPs" configuration, to allow trusted bots easier harvesting access to DSpace (for SEO or similar) and also limit SSR to only "on campus" or other similar IP ranges.

What special behaviors would trusted bots possibly have?

1. Trusted bots/IPs would always have SSR enabled (to allow for harvesting). This would also allow sites to potentially disable SSR site-wide, blocking out any un-trusted bots which cannot parse Javascript. Other known IPs (like "on campus" users) would also have SSR enabled.
2. (Optionally) Trusted bots/IPs might be treated differently by our ui > rateLimiter. Or, perhaps there's just a separate configuration for trusted bots in the rateLimiter, to allow sites to potentially loosen rules for trusted bots and tighten rules elsewhere.
3. Trusted bots would be known, verifiable bots which could also be configured to serve cached SSR content (see cache settings) if necessary for performance.

How would trusted bots be identified?

Trusted bots would need to be identified by IP address/range. Identifying bots by user-agent is no longer recommended practice because it's trivial to fake.

DSpace could populate an initial list of trusted bots from trusted sources such as:

• https://www.gstatic.com/ipranges/goog.json -> IPs of all Googlebots
• https://www.bing.com/toolbox/bingbot.json -> IPs of all Bing bots
• https://duckduckgo.com/duckduckbot.json -> IPs of all DuckDuckGo bots
• Any bot IPs which are auto-populated would also need to be auto-updated from the trusted sources on a regular basis, e.g. once per week. These lists won't change frequently, but IPs may be added/removed.

Individual DSpace sites should also have a configuration (or similar) which allows them to add/remove/update additional "trusted bots". This allows sites to give access to other trusted crawlers / harvesters, or maybe remove access if a previously-trusted bot becomes problematic.

How will this alleviate issues with aggressive bots?

By providing a way to identify the good/trusted bots, DSpace sites could then turn off SSR without any impact on SEO (because sites could configure SSR to only be used for trusted bots).

Turning off SSR helps to deter aggressive bots since many of those bots currently attempt to harvest/parse HTML. Even if they do parse Javascript, they'd do all that processing on the client-side, which would not overwhelm the node server where dspace-angular runs.

Describe the solution you'd like

While I don't have an exact design in mind, we may wish to consider if a list of "trusted IPs" is best kept on the backend or fronted.
* If kept on the frontend, it'd likely need to be a large JSON file which can be parsed at runtime (and rebuilt on demand from configs or similar).
* If kept on the backend, we'd have more flexibility of keeping it in the database or configuration (or a bit of both). However, we'd still need to pass the raw data to the frontend via JSON.

Rough Implementation Proposal

For simplicity, consider storing the list of trusted IP addresses as a JSON file, likely on the backend (as it will need a script to regularly update it, and most of our regular cronjob-style scripts are on the backend).

The JSON file we store locally should use the JSON-Based format for publishing IP Ranges of Automated HTTP Clients, as that's the same format used by all search engines.

1. Create a new set of configurations in dspace.cfg (or a similar *.cfg file) which is structured similar to this:

   # Configure trusted IP sources, which are all defined below.
   # Additional sources can be added by adding a new "webui.trusted-ipranges.source.*" config for each
   webui.trusted-ipranges.sources = google, bing, duckduckgo, custom
   
   # Google source is a JSON file which will be parsed
   # (NOTE: All these JSON files are the same format for every search engine)
   webui.trusted-ipranges.source.google = https://www.gstatic.com/ipranges/goog.json
   
   # BingBot source
   webui.trusted-ipranges.source.bing = https://www.bing.com/toolbox/bingbot.json
   
   # DuckDuckGo source
   webui.trusted-ipranges.source.duckduckgo = https://duckduckgo.com/duckduckbot.json
   
   # Custom source is a list of IP addresses which may correspond to on-campus users or similar
   webui.trusted-ipranges.source.custom = 192.168.0.0/24, 192.168.2.2
   
   # Additional sources can be added by just adding new "webui.trusted-ipranges.source.*" configs
   # The text after "source" allows you to label the meaning behind the IPs.
   # webui.trusted-ipranges.source.custom2 = 192.168.5.5, 192.168.6.6
   
   # Schedule for how frequently the trusted IP ranges should be regenerated via config
   # This example shows it running at 2:15AM every Saturday
   # When commented out, this feature would be disabled
   # webui.trusted-ipranges.cron = 0 15 2 ? * SAT
   

2. The DSpace backend needs a script (e.g. ./dspace update-trusted-ipranges) which can loop through the webui.trusted-ipranges.sources and gather ALL IP ranges together into a single, large JSON file (named something like [dspace]/config/trusted-ipranges.json).
   • This script should have a "cron" setting (see example above), allowing it to be run on a semi-regular basis, e.g. about once per week, because the IPs of search engines don't change frequently. The script should overwrite the existing JSON file each time it runs.
   • Think of this script as similar to the generate-sitemaps script which auto-updates Sitemaps on a regular basis also using a "cron" configuration.
3. The single combined JSON file would use the standard JSON format used by all search engines, and look similar to below. Notice how the "service" in this JSON format would list the source of the IP address/range. That can allow the UI to potentially have different configurations for different services (see examples below).

   {
      "creationTime": "2024-01-03T10:00:00.121331",
      "prefixes": [
        {
          "ipv4Prefix": "157.55.39.0/24",
          "service": "bing"
        },
        {
          "ipv4Prefix": "8.8.8.0/24",
          "service": "google"
        },
        {
          "ipv4Prefix": "104.43.54.127/32",
          "service": "duckduckgo"
        },
        {
          "ipv4Prefix": "192.168.0.0/24",
          "service": "custom"
        },
        ...
   }
   

4. The backend will need to make the trusted-ipranges.json available to the frontend, perhaps proxied similar to how sitemap files generated on the backend are available in the frontend.
   • This might mean making this JSON file visible publicly. However, if we determine that is a security or privacy risk we likely could limit its access only to the server where the UI is running, as this file primarily needs to be visible to the server.ts file in the UI.
5. The User Interface will need to be updated to request this trusted-ipranges.json and parse it to determine whether SSR is enabled. An additional SSR config will need to be added similar to this:

   ssr: 
     # When this is enabled, only trusted IPs listed in `trusted-ipranges.json` will trigger SSR.
     enableTrustedIPRanges: true
     # (OPTIONAL) we could even make it configurable based on service type in "trusted-ipranges.json"
     # This example would only enable SSR for "google" and "custom" services in the "trusted-ipranges.json" file.
     enableTrustedIPRangeServices:
       - service: "google"
       - service: "custom"
   

6. (OPTIONAL) We optionally could update the "cache" settings to customize which cache is used for which service.

   cache: 
     serverSide:
       # This example shows us configuring that the "google" and "bing" IPs should all use the "botCache"
       botCache:
         max: 1000
         trustedIPRangeServices:
           - service: "google"
           - service: "bing"
       # But, the "custom" IPs should use the "anonymousCache" instead
       anonymousCache:
         max: 100
         trustedIPRangeServices:
           - service: "custom"
   

7. Finally, obviously in the UI's server.ts, we'd need a script/library that can quickly determine if the current user IP matches any of those in the trusted-ipranges.json. If a match is found, then SSR is triggered (and caching is triggered, if enabled). If not, then it falls back to CSR only.

Describe alternatives or workarounds you've considered

DSpace doesn't currently have a solution that lines up with this idea:

• DSpace can support only enabling SSR for some pages. But, that's a global config and doesn't allow you to change the settings per IP
• DSpace can support rateLimiting as well, but again it's a global config. No way to have different settings for different IP ranges
• DSpace also supports basic cached of SSR with a bot-specific configuration. But, that bot-specific configuration uses the isbot library which tries to identify bots based on user-agent (which is less reliable and may not work at all for highly aggressive bots)

Additional information

This issue is related to #4565 and comes out of discussions with the Google Scholar team on Sept 8, 2026.

[paulo-graca]

I don't think managing the network or the traffic volume should be a concern of this software. Adding those, it will add more complexify to DSpace configurations and to also for troubleshooting issues.

Other solutions like Anubis could be considered.

[dheles]

The harsh reality is that waves of extremely high (and hard to manage) bot traffic have become the new normal, and that situation does not look like it will change any time soon. It is becoming harder and more resource- and cost-prohibitive to host DSpace and other repositories. Any approaches that might help folks keep their DSpace instances up and running should be considered, as they will likely be of use to someone, somewhere, under some conditions. Clearly, folks are already finding it necessary to turn off SSR to deal with this non-hypothetical problem, so this solution seems like it would be a valuable mitigator to minimize the downsides of this defense.

[pnbecker]

I think we are trying to circumvent a problem that is caused by DSpace's frontend needing too many resource. We should investigate in a central cache for dspace-angular and in a much smaller footprint regarding CPU and memory consumption. I know that this is not easy and we already tried to go this way. But, I don't like the idea to switch off SSR for most requests.

We introduced SSR for bots and for accessibility. The idea of Open Access is that content should be crawlable and indexable (compare the declaration of the budapest open access initiative from 2002). If SSR is to intensive for servers, we should discuss if there is a way to pregenerate some of it's contents or the whole pages and store them on disk. We need to come to the point, that we can serve html without using so many resources as we currently do.

[pnbecker]

The kernel.org maintainers stated in a blog post:

In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity.

source: https://people.kernel.org/monsieuricon/creepy-crawlies

To me, it means we should look which features we have for unauthenticated requests, for example on community, collection or item views, but we should not switch of SSR if it really is still needed for accessibility and for bots.

While I hate the trouble bots make, I think it is important to do the best that we can to provide our contents (see my previous comment -> Open Access).

[paulo-graca]

If SSR is to intensive for servers, we should discuss if there is a way to pregenerate some of it's contents or the whole pages and store them on disk.

Because we host DSpace instalations, In order to display specific and static content for each repository, like in the image:

we've created an Angular component used to load and display static HTML content dynamically, typically news content on the home page. It behavior is to Download the file as plain text using Angular’s HttpClient and the downloaded HTML is injected into the template element.

this.divBinder.nativeElement.innerHTML = html;

that HTML request is cached (outside DSpace).

[vidiecan]

I think there are really two separate questions here:

1. Should repositories remain openly accessible to humans and machines? Yes.
2. Should repository operators have the option to avoid spending SSR resources on every unknown automated client? I think they should.

I would support implementing this as a configurable feature.

One possible policy would be to provide SSR to verified/trusted crawlers that benefit from it, while allowing other automated traffic to fall back to CSR. Google Scholar is an obvious example where SSR matters and the crawler can be verified.

Technically, the benefit is straightforward: an SSR cache miss requires the Node process to execute Angular and render the page on behalf of the requester. With CSR, that rendering work moves to the client. REST/API, database and bandwidth costs remain, so this is not a complete bot mitigation mechanism, but it removes one expensive part of the request path.

And this is not really about trying to hide content from determined crawlers. Modern crawlers that genuinely want the content can execute JavaScript using a headless browser and render the application themselves. The difference is that DSpace no longer has to spend its own SSR resources doing that work for every unknown requester.

This would not replace better caching, rate limiting, or further SSR performance work. It would simply give repository operators another useful control when automated traffic is causing operational problems.

There are trade-offs, including legitimate but unknown crawlers receiving CSR instead of SSR, which is why I think this makes most sense as an optional, configurable feature.

[pnbecker]

I still like the idea of static generated pages, that @kshepherd suggested in #3183. I see a huge potential in having pre-generated static pages for unauthorized users and dynamic pages for authorized users.

[saschaszott]

Can we take any additional ideas from the lessons learned in this paper? 👉 https://journal.code4lib.org/articles/18489

[saschaszott]

@tdonohue , In my view, DSpace currently has an untapped opportunity that could noticeably improve performance and, in particular, reduce the number of requests to the database and Solr (and therefore the load on RAM and the file system) caused by bots.

Unlike the SSR mechanism (in the DSpace frontend), we do not currently cache REST API requests. API requests to item endpoints, especially for in_archive items, often return unchanged data over long periods and do not need to be recalculated by the backend each time.

Combined with limiting browsing and search options for unauthenticated users, caching REST API requests could significantly reduce the load on the DSpace backend.

In this context, we should investigate DSpace issue #9552 further, as it may negatively affect REST API request caching.