String matching against non-UTF8/UTF16 codepage

[Still34]

I'm currently looking to create an additional rule that matches against executable compiled with Easy Programming Language; however, capa does not appear to support matching non-UTF8/UTF16 codepage, and the anchors that most reliably match against EPL is with its 936-codepage strings - this means I would have to fallback to using bytes, which would then have to reduce the broad scope of file down to either unsupported or function: neither of which is ideal. Any idea what would be the best approach around this issue?

Current draft

rule:
  meta:
    name: compiled with easy programming language
    namespace: compiler/epl
    authors:
      - still@teamt5.org
    description: compiled using Easy Programming Language (易语言)
    scopes:
      static: file
      dynamic: file
    examples:
      - 569a3342510f4a89639a7fe7803780d5487ccec3b6163e0cbbab6bb268e24850
  features:
    - 3 or more:
      - bytes: CB F9 CC E1 B9 A9 B2 CE CA FD B5 C4 CA FD BE DD C0 E0 D0 CD B2 BB B7 FB BA CF D2 AA C7 F3 00 = 所提供参数的数据类型不符合要求
      - bytes: CA FD D7 E9 B3 C9 D4 B1 D2 FD D3 C3 CF C2 B1 EA B1 D8 D0 EB B4 F3 D3 DA B5 C8 D3 DA 31 00 = 数组成员引用下标必须大于等于1
      - bytes: B5 F7 D3 C3 44 4C 4C C3 FC C1 EE BA F3 B7 A2 CF D6 B6 D1 D5 BB B4 ED CE F3 A3 AC CD A8 B3 A3 CA C7 44 4C 4C B2 CE CA FD CA FD C4 BF B2 BB D5 FD C8 B7 00 = 调用DLL命令后发现堆栈错误，通常是DLL参数数目不正确
      - bytes: 70 72 6F 67 72 61 6D 20 69 6E 74 65 72 6E 61 6C 20 65 72 72 6F 72 20 6E 75 6D 62 65 72 20 69 73 20 25 64 2E 20 0D 0A 25 73 0D 0A B4 ED CE F3 CE BB D6 C3 3A 25 64 2C 25 64 00 = program internal error number is %d. \r\n%s\r\n错误位置:%d,%d
      - bytes: C9 EA C7 EB 25 64 D7 D6 BD DA C4 DA B4 E6 CA A7 B0 DC 2E 0D 0A D5 BB D0 C5 CF A2 3A 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 30 78 25 30 38 58 0D 0A 00 = 申请%d字节内存失败.\r\n栈信息:\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n0x%08X\r\n

[williballenthin]

we only support utf-8/utf-16le because we don't have a robust way to detect other character encodings. if you have a lot of experience in this area, we're happy to entertain ideas.

[Still34]

From what I've seen there's really not a whole lot of robust way to identify whether something is a legitimate string for Simplified Chinese GBK. Whilst I currently do have a script that flags possible ones, there will always be false positives for lead to complete gibberish - i.e., non-valid Chinese strings.

[williballenthin]

i suppose we could support byte matching at file scope as a scan for those bytes. we could use yara, or regex/aho-corasick, to do this somewhat efficiently.

in the past we haven't done this, since yara is usually better for scanning file bytes, but perhaps to keep things in the ecosystem we could consider it. however, we'd want to discourage people from trying to do more yara-like things with capa file byte scanning (like looking for mnemonics or whatever).

[Still34]

I certainly don't think the project should be supporting or encouraging mnemonic specific byte scans like instructions-based YARAs do, and I understand this is a bit of a can of worm situation - though I think bytes scanning not being supported on a file level has always been a bit unintuitive, especially if the bytes are present in the file but never referenced directly by one or more function.