Skip to content

add results for tencent/EVIE-8B and tencent/EVIE-4.5B - #715

Merged
KennethEnevoldsen merged 2 commits into
mainfrom
add-evie
Sep 20, 2026
Merged

KennethEnevoldsen merged 2 commits into
mainfrom
add-evie

Conversation

@KennethEnevoldsen

Copy link
Copy Markdown
Contributor

Note see embeddings-benchmark/mteb#5451 for discussion on implementation

closes embeddings-benchmark/mteb#5451

Checklist

  • My model has a model sheet, report, or similar
  • My model has a reference implementation in mteb/models/model_implementations/, this can be as an API. Instruction on how to add a model can be found here
    • No, but there is an existing PR ___
  • The results submitted are obtained using the reference implementation
  • My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
  • I solemnly swear that for all results submitted I have not trained on the evaluation dataset including training splits. If I have, I have disclosed it clearly.

@KennethEnevoldsen
KennethEnevoldsen marked this pull request as draft September 15, 2026 14:22
@github-actions

github-actions Bot commented Sep 15, 2026

Copy link
Copy Markdown

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: tencent/EVIE-4.5B, tencent/EVIE-8B

Results for tencent/EVIE-4.5B

task_name tencent/EVIE-4.5B Max result Model with max result In Training Data
Vidore3NuclearRetrieval .564 .549 vultr/VultronRetrieverCore-Qwen3.5-4.5B False
Vidore3TelecomRetrieval .735 .733 webAI-Official/webAI-ColVec1.1-8b False
Average .650 .641 nan -

Model have high performance on these tasks: Vidore3TelecomRetrieval,Vidore3NuclearRetrieval


Results for tencent/EVIE-8B

task_name tencent/EVIE-8B Max result Model with max result In Training Data
Vidore3NuclearRetrieval .566 .549 vultr/VultronRetrieverCore-Qwen3.5-4.5B False
Vidore3TelecomRetrieval .743 .733 webAI-Official/webAI-ColVec1.1-8b False
Average .654 .641 nan -

Model have high performance on these tasks: Vidore3TelecomRetrieval,Vidore3NuclearRetrieval


@officea1t

Copy link
Copy Markdown
Contributor

Hi @KennethEnevoldsen, thanks again for running and adding the private Nuclear/Telecom results! 🙏

This PR is marked ready for review and closes #5451. Whenever you (or @Samoed) have a moment, would it be possible to merge it so the ViDoRe V3 entry is complete (8 public domains + the 2 private tasks)? Happy to help if anything else is needed on our side. Thank you!

@KennethEnevoldsen
KennethEnevoldsen merged commit eddc167 into main Sep 20, 2026
4 checks passed
@KennethEnevoldsen
KennethEnevoldsen deleted the add-evie branch September 20, 2026 19:36
@KennethEnevoldsen

Copy link
Copy Markdown
Contributor Author

Sorry @officea1t I see that I wasn't quite clear in my communication. I was waiting for a review from you guys stating that you were happy to merge it (I assume you are from the message)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation request — tencent/EVIE-8B and tencent/EVIE-4.5B on ViDoRe(v3) private tasks

2 participants