无奈 - 何事西风不待人

Stop crawling my code repositories

2026-07-09T19:00:00.000Z

I don't know if it's related to the prevalence of language-model-based automation tools, but for the past few years, I have been seeing logs of page loads of blame pages and file-at-commit pages for my code repositories at Forgejo and Phorge. Their IP was randomly distributed, and the rate goes up to 10 requests per second. This does not look normal for our small site, and it made the load average of the server hike up to 40 (!).

For Forgejo, the main problem is it invokes git every time it receives such request, and that git command is very costly. For Phorge, the main cost is invoking the pygmentize code highlighter.

So I decided to rate-limit the requests. I started with 5 requests per second limit, and paired that with fail2ban. The load went down, but it wasn't enough for us to maintain the stability of the server. Normal requests still took unreasonable time and it could time out at any moment. So I set the rate-limit to a more aggressive 1 request per second. Good enough for some time, but still problematic, as the rate-limit often unnecessarily hurts the experience of legit users like myself, and I have seen myself banned by fail2ban because I was the only one requested twice in a row with the same IP. Crawlers, on the other hand, always use hundreds of thousands of different IPs.

So this needs to end. One way to do it is to let logged-in users view the pages as usual while keeping out non-logged-in crawlers.

You probably want to interject for a moment, saying "Why not proof of work?" And the answer is, I really hate the idea of using a machine to check whether someone is human. So things like Anubis or Go Away is none of interest for me.

And I let the shit generator to generate such a config for me. The idea is to use the session cookie: ask the reverse proxy to access a page that requires authentication, but is much less costly than rendering the file-at-commit and blame pages.

For Forgejo, the page could be /user/settings. It will return 200 if the user is logged in, but non-logged-in users will get a 303 redirect to the login page. The result is the nginx configuration below:

  location = /_auth {
    internal;
    proxy_method HEAD;
    proxy_pass http://forgejo/user/settings;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header Host r.lily-is.land;
    proxy_set_header Cookie $http_cookie;
    proxy_intercept_errors on;
    error_page 301 302 303 307 =401 /_auth_fail;
  }

  location @login_redirect {
    add_header Content-Type text/html always;
    return 401 "<html><body><h1>Unauthorized</h1><p>Due to spam, we have limited file at commit and blame pages to logged-in users. Please <a href='https://r.lily-is.land/user/login'>login</a>.</p></body></html>";
  }

  location ~ ^/[^/]+/[^/]+/(src|commits|raw|blame)/commit {
    auth_request /_auth;
    error_page 401 = @login_redirect;
    proxy_pass http://forgejo;
    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

For Phorge, the problem is a little bit more tricky, because I cannot find something that return 2xx if and only if the user is logged in. Quite the opposite, many pages, like the settings page, return 200 if and only if the user is not logged in. So I went on with composing another mini-server that analyze the result and convert the status code to my needs.

#!/usr/bin/env python3
import http.server, urllib.request

PHORGE = "http://......" # intranet address of phorge instance
PHORGE_HOST = "iron.lily-is.land"

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        cookie = self.headers.get("Cookie", "")
        req = urllib.request.Request(f"{PHORGE}/settings/", method="HEAD")
        req.add_header("Host", PHORGE_HOST)
        req.add_header("Cookie", cookie)
        try:
            resp = urllib.request.urlopen(req)
            print('Response:', resp)
            self.send_response(200 if resp.url.startswith(f"https://{PHORGE_HOST}/settings/user/") else 401)
        except Exception as e:
            print('Exception:', e)
            self.send_response(401)
        self.end_headers()

http.server.HTTPServer(("0.0.0.0", 9090), Handler).serve_forever()

The reasoning is as follows: When accessing HEAD /settings/, if the session cookie is valid for a logged-in user, it redirects to <Public-Address>/settings/user/<Username>/, otherwise it, well, it can behave in a couple of ways, like returning 200 (with a login form if it is a GET request) or 500 (why?). It is worth notice that the session cookie has phusr=<Username>, and I tried requesting the endpoint with a bad cookie like phsid=xxx; phusr=tusooa (where xxx is a valid session id but with one letter different, because Phorge tends to check length and format before actual checks!), and it will not make the redirect to /settings/user/tusooa/, so this method is safe.

And the nginx configuration:

  location = /_phorge_auth {
    internal;
    proxy_pass http://logincheckserver/; # Must have trailing slash, otherwise it requests GET /_phorge_auth instead of GET /
    proxy_pass_request_body off;
    proxy_set_header Cookie $http_cookie;
    proxy_set_header Content-Length "";
  }

  location ~ ^/diffusion/[^/]+/(history|browse) {
    auth_request /_phorge_auth;

    proxy_pass http://phorge;
    proxy_redirect off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Protocol $scheme;
    proxy_set_header X-Url-Scheme $scheme;
    error_page 401 = @not_logged_in;
  }

  location @not_logged_in {
    add_header Content-Type text/html always;
    return 401 "<html><body><h1>Unauthorized</h1><p>Due to spam, we have limited file at commit and blame pages to logged-in users. Please <a href='/auth/start/'>login</a>, use the <a href='https://r.lily-is.land/'>Forgejo instance</a>, or better yet, clone the repository for your local use.</p></body></html>";
  }

Hopefully I can get rid of the crawlers soon. Last time I disabled these endpoints completely on Forgejo and returned 403 for them, the rate of crawler requests decreased significantly.

Fix firefox address bar clickSelectsAll bug (Spoiler: no re-compilation required.)

2020-10-01T20:15:56.000Z

This is last updated 2026,5,29 (5) to reflect new changes for Firefox >= 201. View update history. This post has been updated 4 times to adapt to upstream changes.

A couple of months ago, I opened my Firefox as usual, I found out that when I click on the address bar, it dared to select everything in it as if any creepy browser would do! The first thing I did is to check the clickSelectsAll pref -- what I always do first after getting a new install of Firefox on Windows is setting this pref to false, and setting doubleClickSelectsAll pref to true: that is the default behaviour for Firefox on GNU/Linux... that time at least...

Much to my disappointment, the pref are setting correctly, but nothing works. I tried to search for this, but without success, maybe since this bug was so new, or I was to silly to apply the right keywords (huh, maybe the latter one?). So I created an account on Mozilla's bugtracker just to open a bug. At the same time I switched back to the ESR version of Firefox, which rid me (till recently) of this stupid address bar but still granted me security updates. A few days ago, however, Firefox ESR was unfortunately eventually shipped with this bug, calling for a solution.

Some more background

Here I will add more background because I need to complain a lot (>w<). If you would like to see the solution directly, jump there.

Someone on bugzilla pointed out that the bug had been raised by someone else. Following that bug, I saw that the developer was being pretty toxic and just marked it as WONTFIX, despite many people criticized the buggy behaviour and regarded the original behaviour as signature of Firefox. I finally switched from Ubuntu to Gentoo after the new ESR, as I would like to have more control of my software.

Yesterday I talked to tiar about the freedom of users to make software behave the way they want, without necessarily being a developer. I complained that many free/libre software does not excel in this aspect, using Firefox >=75 as an example.

The thing Firefox developers does, by removing the clickSelectsAll prefs, is actually somewhat limiting users' freedom, as customization becomes harder and harder. This kind of freedom has been brought up by a KDE person as a goal called "freedom out of the box." In short, it aims to make changing the behaviour of a program as easy as just using it, probably unifying the interface where a program is used and where a program is customized.

And that is reasonable: no one should have to spend quite a long time reading an extraordinarily large code base just to make a tiny little change that would significantly improve their lives. We will simply not have that much time. I know tweaking system is fun... (should I say addictive?) but still, instant result is always a good-to-have.

The solution

For Firefox, we do have a solution without needing to recompile. That is, to change the omni.ja pack in the Firefox distribution. Based on that I managed to restore the old good urlbar behaviour.

For my firefox-bin on Gentoo, the relevant file is at /opt/firefox/browser/omni.ja. (In Firefox v146 with https://phabricator.services.mozilla.com/D270553#change-373rnO3LTLeb, the file was once shortly in omni.ja instead of browser/omni.ja.)

First make a copy of that file:

cp -v /opt/firefox/browser/omni.ja omni.ja.orig

Then unzip the file:

mkdir omni && unzip -d omni /opt/firefox/browser/omni.ja

It may return 2, but does not matter.

Then edit the chrome/browser/content/browser/urlbar/UrlbarInput.mjs. In the _on_mousedown function:

Change

this.#preventClickSelectsAll = this.focused;

this.#preventClickSelectsAll = true;

to get the (no) clickSelectsAll behaviour.

Before

      // Do not suppress the focus border if we are already focused. If we
      // did, we'd hide the focus border briefly then show it again if the
      // user has Top Sites disabled, creating a flashing effect.
      this.view.autoOpen({

, add:

     // doubleClickSelectsAll
     if (event.detail == 2) {
       this.select();
       event.preventDefault();
       break;
     }

to get the doubleClickSelectsAll behaviour.

Note that this will not put the url into primary selection upon double click. This can be useful sometimes, if you just would like to replace the current url with the one currently in primary selection. If you would like to put the url into primary selection upon double click, which completely restores the original behaviour, change this.select(); to this.inputField.select();.

I have also made a patch (command diff -Naru omni-orig omni-new). Download the patch here.

There is also the search bar to change. The original Superuser answer suggests this, but as I do not use the search bar, I have not tried it out, nor do I know what the original behaviour of it was.

sed -i 's/this\._preventClickSelectsAll = this\._textbox\.focused;/this._preventClickSelectsAll = true;/'  omni/chrome/browser/content/browser/search/searchbar.js

After editing, we need to re-pack the files:

cd omni && zip -0DXqr ../omni.ja * && cd -

And replace the original one:

cp -v omni.ja /opt/firefox/browser/omni.ja

You may need to change the ownership of the newly-created archive:

chown [U]:[G] /opt/firefox/browser/omni.ja

where [U] and [G] refers to the owner and group of the original file (we made a backup at omni.ja.orig).

To ensure everything takes effect, make a file called .purgecaches to make Firefox aware of the changes:

touch /opt/firefox/browser/.purgecaches

Okay, from now on, we shall no longer suffer from the clickSelectsAll bug.

Also, you may want to have additional configurations, such as disabling the pop-up list upon clicking address bar or gettind rid of the address bar expanding effect when clicked upon.

短代码(一个马后炮)

2018-04-29T06:58:57.000Z

没赶上报名截止日期，一直 not invited。就很难过。

E 的题目描述是这样的。

Description

tz学姐做题很忙，有一些简单的问题需要你帮忙解决：给定三个序列A,B,C和一个整数X, 现在需要找到三个数Ai，Bj，Ck，满足Ai+Bj+Ck = X.

Input

第1行三个整数 L, N, M；

第2行L个整数代表序列A

第3行M个整数代表序列B

第4行N个整数代表序列C

第5行1个整数S代表有S个X。

接下来的S行每行一个整数X

1<=L, N, M<=500, 1<=S<=1000. 所有整数均在32位整数范围内

Output

对于S个询问计算X对应的公式是否能被满足。如能满足，输出 "YES", 否则输出 "NO"。

Sample Input

Sample Output

NO
YES
NO

喵

我当时给出的代码是这样的。

#include <iostream>
#include <algorithm>
using namespace std;
int main() {
  int l,m,n;
  cin>>l>>m>>n;
  int a[l],b[m],c[n];
  for(int&p:a)cin>>p;
  for(int&p:b)cin>>p;
  for(int&p:c)cin>>p;
  sort(c,c+n);
  cin>>n;
  for(int p;cin>>p;){
    for(int q:a)
      for(int r:b)
        if(binary_search(c,c+n,p-q-r)) {
          cout<<"YES\n";
          goto l;
        }
    cout<<"NO\n";
  l:1;
  }
}
//292

由于我不能提交，无法判断它是否超时了。查了一下网上的做法，是把 a 和 b 加起来存到另一个数组里，然后对这个新的数组排序查找。

仔细分析一下就是：

我的方法，排序用了 O(n*log n)，查找 O(n**2*log n)。

另一个做法，排序 O(n**2*log n)，查找 O(n*log n)。

看上去好像正好抵消了没什么区别。问题在于查找是多次查找，我们希望每次查找的时间尽量短。

所以改过的代码是这样的：

#include <iostream>
#include <algorithm>
using namespace std;
int main() {
  int l,m,n,r=0;
  cin>>l>>m>>n;
  int a[l],b[m],c[n],S[l*m]; // GNU 扩展的变长数组
  for(int&p:a)cin>>p;
  // range-for 输入第二组数据，同时将输入的数据和第一组的每一个数字相累加，结果存到 `S` 里。
  for(int&p:b) {
    cin>>p;
    for(int&q:a)S[r++]=p+q;
  }
  for(int&p:c)cin>>p;
  sort(S,S+r);
  cin>>n; // 丢弃个数信息，因为不需要
  for(int p;cin>>p;){
    for(int q:c)
      if(binary_search(S,S+r,p-q)){
        puts("YES");
        goto l; // 直接跳到下次输入，不输出 NO
      }
    puts("NO");
  l:1;
  }
}
//311

排名表上最短是345。

但是有什么用呢。我又不能提交(以后一定要提前关注一下)。