Stop crawling my code repositories

2026-07-09T19:00:00.000Z

I don't know if it's related to the prevalence of language-model-based automation tools, but for the past few years, I have been seeing logs of page loads of blame pages and file-at-commit pages for my code repositories at Forgejo and Phorge. Their IP was randomly distributed, and the rate goes up to 10 requests per second. This does not look normal for our small site, and it made the load average of the server hike up to 40 (!).

For Forgejo, the main problem is it invokes git every time it receives such request, and that git command is very costly. For Phorge, the main cost is invoking the pygmentize code highlighter.

So I decided to rate-limit the requests. I started with 5 requests per second limit, and paired that with fail2ban. The load went down, but it wasn't enough for us to maintain the stability of the server. Normal requests still took unreasonable time and it could time out at any moment. So I set the rate-limit to a more aggressive 1 request per second. Good enough for some time, but still problematic, as the rate-limit often unnecessarily hurts the experience of legit users like myself, and I have seen myself banned by fail2ban because I was the only one requested twice in a row with the same IP. Crawlers, on the other hand, always use hundreds of thousands of different IPs.

So this needs to end. One way to do it is to let logged-in users view the pages as usual while keeping out non-logged-in crawlers.

You probably want to interject for a moment, saying "Why not proof of work?" And the answer is, I really hate the idea of using a machine to check whether someone is human. So things like Anubis or Go Away is none of interest for me.

And I let the shit generator to generate such a config for me. The idea is to use the session cookie: ask the reverse proxy to access a page that requires authentication, but is much less costly than rendering the file-at-commit and blame pages.

For Forgejo, the page could be /user/settings. It will return 200 if the user is logged in, but non-logged-in users will get a 303 redirect to the login page. The result is the nginx configuration below:

  location = /_auth {
    internal;
    proxy_method HEAD;
    proxy_pass http://forgejo/user/settings;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header Host r.lily-is.land;
    proxy_set_header Cookie $http_cookie;
    proxy_intercept_errors on;
    error_page 301 302 303 307 =401 /_auth_fail;
  }

  location @login_redirect {
    add_header Content-Type text/html always;
    return 401 "<html><body><h1>Unauthorized</h1><p>Due to spam, we have limited file at commit and blame pages to logged-in users. Please <a href='https://r.lily-is.land/user/login'>login</a>.</p></body></html>";
  }

  location ~ ^/[^/]+/[^/]+/(src|commits|raw|blame)/commit {
    auth_request /_auth;
    error_page 401 = @login_redirect;
    proxy_pass http://forgejo;
    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

For Phorge, the problem is a little bit more tricky, because I cannot find something that return 2xx if and only if the user is logged in. Quite the opposite, many pages, like the settings page, return 200 if and only if the user is not logged in. So I went on with composing another mini-server that analyze the result and convert the status code to my needs.

#!/usr/bin/env python3
import http.server, urllib.request

PHORGE = "http://......" # intranet address of phorge instance
PHORGE_HOST = "iron.lily-is.land"

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        cookie = self.headers.get("Cookie", "")
        req = urllib.request.Request(f"{PHORGE}/settings/", method="HEAD")
        req.add_header("Host", PHORGE_HOST)
        req.add_header("Cookie", cookie)
        try:
            resp = urllib.request.urlopen(req)
            print('Response:', resp)
            self.send_response(200 if resp.url.startswith(f"https://{PHORGE_HOST}/settings/user/") else 401)
        except Exception as e:
            print('Exception:', e)
            self.send_response(401)
        self.end_headers()

http.server.HTTPServer(("0.0.0.0", 9090), Handler).serve_forever()

The reasoning is as follows: When accessing HEAD /settings/, if the session cookie is valid for a logged-in user, it redirects to <Public-Address>/settings/user/<Username>/, otherwise it, well, it can behave in a couple of ways, like returning 200 (with a login form if it is a GET request) or 500 (why?). It is worth notice that the session cookie has phusr=<Username>, and I tried requesting the endpoint with a bad cookie like phsid=xxx; phusr=tusooa (where xxx is a valid session id but with one letter different, because Phorge tends to check length and format before actual checks!), and it will not make the redirect to /settings/user/tusooa/, so this method is safe.

And the nginx configuration:

  location = /_phorge_auth {
    internal;
    proxy_pass http://logincheckserver/; # Must have trailing slash, otherwise it requests GET /_phorge_auth instead of GET /
    proxy_pass_request_body off;
    proxy_set_header Cookie $http_cookie;
    proxy_set_header Content-Length "";
  }

  location ~ ^/diffusion/[^/]+/(history|browse) {
    auth_request /_phorge_auth;

    proxy_pass http://phorge;
    proxy_redirect off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Protocol $scheme;
    proxy_set_header X-Url-Scheme $scheme;
    error_page 401 = @not_logged_in;
  }

  location @not_logged_in {
    add_header Content-Type text/html always;
    return 401 "<html><body><h1>Unauthorized</h1><p>Due to spam, we have limited file at commit and blame pages to logged-in users. Please <a href='/auth/start/'>login</a>, use the <a href='https://r.lily-is.land/'>Forgejo instance</a>, or better yet, clone the repository for your local use.</p></body></html>";
  }

Hopefully I can get rid of the crawlers soon. Last time I disabled these endpoints completely on Forgejo and returned 403 for them, the rate of crawler requests decreased significantly.

forgejo - 何事西风不待人

Stop crawling my code repositories