株洲市建设局网站,深圳最好的营销网站建设公司,wordpress左右滑动页面,网页导航设计步骤文章目录 一、准备1、安装 autoawq2、模型准备 二、量化config.json 文件变化 三、加载量化后模型量化后的输出原始输出对比 四、查看模型的精度1、查看模型卡2、查看 config.json 中的 torch_dtype3、打印模型信息4、model.dtype 未必是模型精度 一、准备
1、安装 autoawq
p… 文章目录 一、准备1、安装 autoawq2、模型准备 二、量化config.json 文件变化 三、加载量化后模型量化后的输出原始输出对比 四、查看模型的精度1、查看模型卡2、查看 config.json 中的 torch_dtype3、打印模型信息4、model.dtype 未必是模型精度 一、准备
1、安装 autoawq
pip install autoawq
pip install transformers4.47.1 使用的较低版本的 transformers如果执行下面代码有问题可以检查 transformers 版本。
目前我的测试 Python 环境为 3.9 2、模型准备
这里以 mistralai/Mistral-7B-Instruct-v0.2 为例
如果下载有问题可以前往模型界面查看是否需要申请权限https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 后面代码会自动下载模型你也可以提前下载模型
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.2如果网络受限可以设置镜像地址到环境变量
export HF_ENDPOINThttps://hf-mirror.com二、量化
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizermodel_path mistralai/Mistral-7B-Instruct-v0.2
quant_path mistral-instruct-v0.2-awq
quant_config { zero_point: True, q_group_size: 128, w_bit: 4, version: GEMM }# Load model
model AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer AutoTokenizer.from_pretrained(model_path, trust_remote_codeTrue)# 查看模型类型
model.dtype
# torch.float32 - 32-bitFP32 # Quantize
model.quantize(tokenizer, quant_configquant_config)# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)print(fModel is quantized and saved at {quant_path})quant_config 也可以写成
from transformers import AwqConfig, AutoConfig
quantization_config AwqConfig(bitsquant_config[w_bit],group_sizequant_config[q_group_size],zero_pointquant_config[zero_point],versionquant_config[version].lower(),
).to_dict()model.model.config.quantization_config quantization_configconfig.json 文件变化
config.json 文件会变成下方的样子
相比原来的文件多出 quantization_config 内容其中 quant_method: awq
{_name_or_path: /home/wx/.cache/huggingface/hub/models--mistralai--Mistral-7B-Instruct-v0.2/snapshots/3ad372fc79158a2148299e3318516c786aeded6c,architectures: [MistralForCausalLM],attention_dropout: 0.0,bos_token_id: 1,eos_token_id: 2,head_dim: 128,hidden_act: silu,hidden_size: 4096,initializer_range: 0.02,intermediate_size: 14336,max_position_embeddings: 32768,model_type: mistral,num_attention_heads: 32,num_hidden_layers: 32,num_key_value_heads: 8,quantization_config: {bits: 4,group_size: 128,modules_to_not_convert: null,quant_method: awq,version: gemm,zero_point: true},rms_norm_eps: 1e-05,rope_theta: 1000000.0,sliding_window: null,tie_word_embeddings: false,torch_dtype: bfloat16,transformers_version: 4.47.1,use_cache: false,vocab_size: 32000
} 原始 config.json
{architectures: [MistralForCausalLM],attention_dropout: 0.0,bos_token_id: 1,eos_token_id: 2,hidden_act: silu,hidden_size: 4096,initializer_range: 0.02,intermediate_size: 14336,max_position_embeddings: 32768,model_type: mistral,num_attention_heads: 32,num_hidden_layers: 32,num_key_value_heads: 8,rms_norm_eps: 1e-05,rope_theta: 1000000.0,sliding_window: null,tie_word_embeddings: false,torch_dtype: bfloat16,transformers_version: 4.36.0,use_cache: true,vocab_size: 32000
} 三、加载量化后模型 from transformers import AutoModelForCausalLM, AutoTokenizer
quant_dir /home/wx/mistral-instruct-v0.2-awq
model AutoModelForCausalLM.from_pretrained(quant_dir, device_mapauto)
tokenizer AutoTokenizer.from_pretrained(quant_dir, trust_remote_codeTrue)prompt Tell me about blackhole.
prompt_templatef{prompt}tokens tokenizer(prompt_template, return_tensorspt).input_ids.cuda()generated_ids model.generate(tokens, do_sampleTrue,temperature0.7,top_p0.95,top_k40,max_new_tokens512)
decoded tokenizer.decode(generated_ids[0])
print(decoded)量化后的输出
GPU 占用4550MiB
s Tell me about blackhole.A black hole is a region in space where the gravitational pull is so strong that nothing, not even light, can escape. Its called a black hole because it appears black due to the absence of light emanating from it.Black holes are formed when a massive star collapses in on itself after it has exhausted its nuclear fuel. The collapse causes the star to shrink down to an incredibly small size, creating an incredibly dense object. This object is so dense that its gravity warps space and time around it, forming an event horizon, which is the point of no return. Once anything crosses this event horizon, its pulled into the black hole and cannot escape.Black holes come in different sizes, with the smallest being about the size of a star and the largest being billions of times larger than the sun. The largest black hole that has been discovered is located at the center of the galaxy, and its estimated to be about 40 billion times the mass of the sun.Despite their intimidating name, black holes are not necessarily a threat to us. The closest known black hole to Earth is about 1,600 light-years away, which is far enough that we dont need to worry about being sucked in. However, they are fascinating objects that continue to captivate scientists and the general public alike./s原始输出
mistralai/Mistral-7B-Instruct-v0.2 , GPU 占用21988MiB
s Tell me about blackhole. Ive heard that it is some sort of astronomical thing, but I dont really understand what it is or how it works.A black hole is an extremely dense object in space that has such strong gravitational pull that nothing, not even light, can escape from it once it gets too close. Black holes are formed when a massive star collapses in on itself after it has exhausted its nuclear fuel and can no longer produce the pressure needed to counteract the force of gravity.The boundary around a black hole from which nothing can escape is called the event horizon. This is not a physical boundary that you can see, but rather a theoretical construct based on the laws of physics. Once an object crosses the event horizon, it is considered to be inside the black hole itself.Black holes are not completely black, as they do emit some form of radiation, but they appear black because they absorb all the light that falls on them. This is due to the fact that the intense gravitational pull causes the surface of the black hole to be at a temperature so hot that it emits very little visible light.Black holes can vary in size, from small ones that are only a few times the mass of the sun, to supermassive black holes that can be millions or even billions of times the mass of the sun. The supermassive black holes are thought to be at the center of most, if not all, galaxies, including our own Milky Way.Despite their fearsome reputation, black holes are not a threat to us here on Earth, as they are typically located billions of light-years away. However, they are fascinating objects of study for astronomers and physicists, who continue to learn new things about them and their role in the universe./s对比
原始4bit 量化后占用磁盘大小14G3.9GGPU 占用21988MiB4550MiB 4.8倍 四、查看模型的精度
对于一个模型我们想知道原始的精度是多少可以用下面几种方式
1、查看模型卡
如https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 右边的 Safetensors 信息 2、查看 config.json 中的 torch_dtype
torch_dtype: bfloat16, 3、打印模型信息
from transformers import AutoTokenizermodel_path mistralai/Mistral-7B-Instruct-v0.2# Load model
model AutoAWQForCausalLM.from_pretrained(model_path)for name, param in model.named_parameters():print(f{name}: {param.dtype})break # 只打印第一个权重的数据类型4、model.dtype 未必是模型精度
上述模型model.dtype 打印的结果为 torch.float32表示模型当前是以 32-bit 浮点数FP32精度加载的。 config.json 中的 torch_dtype: bfloat16表示模型设计时支持或推荐使用 bfloat16 精度但实际加载时可能由于环境 或 代码设置 未启用 bfloat16。 2025-03-08六
