https://github.com/NLPdeveloper/NER

IE-Model

1B

https://blog.csdn.net

FewCLUE.M

1B

https://github.com/GTCOM-NLP/JoveBM

roberta-xxlarge

1B

https://github.com/huggingface/transformers

十亿级预训练模型，数百G高质量语料；基于5种mask策略训练。

1B

https://github.com/huggingface/transformers

1B模型，span解码

1B

https://github.com/GeorgeLuImmortal/DML_CLUENER/tree/main

Deep Mutual Learning + nezha large

324M

https://github.com/CLUEbenchmark/CLUENER2020

-

-

https://github.com/liuwei1206/LEBERT

使用知识图谱+le+nezha

100m

https://github.com/ymcui/MacBERT

MacBERT-large

324M

https://github.com/autoliuweijie/K-BERT

voting

1M

https://mp.weixin.qq.com/s/CavGiy1Rz0MJVtcxXdSn0A

联合团队致力于将知识融入预训练模型，进而充分发挥已有参数下的模型潜力。模型结构方面，“神农”基于 Transformer 架构，仅包含十亿级的参数量。从数据量来看，“神农”以数百 GB 级的平文本做基石，涵盖百科、论坛博客、新闻、财经等众多领域的高质量文本。

基于知识的预训练模型

https://github.com/GTCOM-NLP/JoveBM

roberta-xxlarge

1B

https://github.com/FhzzNlp/fhzzNer

众智NLP

3B

https://github.com/GTCOM-NLP/JoveBM

roberta-xxlarge

1B

https://github.com/huawei-noah/Pretrained-Language-Model

哪吒

300M

https://blog.csdn.net

FewCLUE.M

1B

https://arxiv.org/abs/2105.07148

Lexicon enhanced kbert

1000000

https://mp.weixin.qq.com/s/HQL0Hk49UR6kVNtrvcXEGA

摩天预训练大模型；十亿级别参数量，1TB 高价值数据，优化Masked language model遮蔽方案，研发一种相对位置编码方案，大规模/大batch预训练模型训练能力优化

1B

https://github.com/google-research/bert

roberta-base

1000000

https://github.com/PaddlePaddle/PaddleNLP

FewCLUE.M

1B