一.前言
近期， ChatGLM-6B 的第二代版本ChatGLM2-6B已经正式发布，引入了如下新特性：
①. 基座模型升级，性能更强大，在中文C-Eval榜单中，以51.7分位列第6；
②. 支持8K-32k的上下文；
③. 推理性能提升了42%；
④. 对学术研究完全开放，允许申请商用授权。
目前大多数部署方案采用的是fastapi+uvicorn+transformers，这种方式适合快速运行一些demo，在生产环境中使用还是推荐使用专门的深度学习推理服务框架，如Triton。本文将介绍我利用集团9n-triton工具部署ChatGLM2-6B过程中踩过的一些坑，希望可以为有部署需求的同学提供一些帮助。
二.硬件要求
部署的硬件要求可以参考如下：
| 量化等级 | 编码 2048 长度的最小显存 | 生成 8192 长度的最小显存 |
|---|---|---|
| FP16 / BF16 | 13.1 GB | 12.8 GB |
| INT8 | 8.2 GB | 8.1 GB |
| INT4 | 5.5 GB | 5.1 GB |
我部署了2个pod，每个pod的资源：CPU（4核）、内存（30G）、1张P40显卡（显存24G）。
三.部署实践
Triton默认支持的PyTorch模型格式为TorchScript，由于ChatGLM2-6B模型转换成TorchScript格式会报错，本文将以Python Backend的方式进行部署。
1. 模型目录结构
9N-Triton使用集成模型，如上图所示模型仓库(model_repository), 它内部可以包含一个或多个子模型（如chatglm2-6b)。下面对各个部分进行展开介绍：
2. python执行环境
该部分为模型推理时需要的相关python依赖包，可以使用conda-pack将conda虚拟环境打包，如python-3-8.tar.gz。如对打包conda环境不熟悉的，可以参考 https://conda.github.io/conda-pack/。然后在config.pbtxt中配置执行环境路径：
1parameters: { 2 key: "EXECUTION_ENV_PATH", 3 value: {string_value: "$$TRITON_MODEL_DIRECTORY/../python-3-8.tar.gz"} 4} 5
在当前示例中，$$TRITON_MODEL_DIRECTORY="$pwd/model_repository/chatglm2-6b"。
注意：当前python执行环境为所有子模型共享，如果想给不同子模型指定不同的执行环境，则应该将tar.gz文件放在子模型目录下，如下所示：
同时，在config.pbtxt中配置执行环境路径如下：
1parameters: { 2 key: "EXECUTION_ENV_PATH", 3 value: {string_value: "$$TRITON_MODEL_DIRECTORY/python-3-8.tar.gz"} 4} 5
3. 模型配置文件
模型仓库库中的每个模型都必须包含一个模型配置文件config.pbtxt，用于指定平台和或后端属性、max_batch_size 属性以及模型的输入和输出张量等。ChatGLM2-6B的配置文件可以参考如下：
1name: "chatglm2-6b" // 必填，模型名，需与该子模型的文件夹名字相同 2backend: "python" // 必填，模型所使用的后端引擎 3 4max_batch_size: 0 // 模型每次请求最大的批数据量，张量shape由max_batch_size和dims组合指定，对于 max_batch_size 大于 0 的模型，完整形状形成为 [ -1 ] + dims。 对于 max_batch_size 等于 0 的模型，完整形状形成为 dims。 5input [ // 必填，输入定义 6 { 7 name: "prompt" //必填，名称 8 data_type: TYPE_STRING //必填，数据类型 9 dims: [ -1 ] //必填，数据维度，-1 表示可变维度 10 }, 11 { 12 name: "history" 13 data_type: TYPE_STRING 14 dims: [ -1 ] 15 }, 16 { 17 name: "temperature" 18 data_type: TYPE_STRING 19 dims: [ -1 ] 20 }, 21 { 22 name: "max_token" 23 data_type: TYPE_STRING 24 dims: [ -1 ] 25 }, 26 { 27 name: "history_len" 28 data_type: TYPE_STRING 29 dims: [ -1 ] 30 } 31] 32output [ //必填，输出定义 33 { 34 name: "response" 35 data_type: TYPE_STRING 36 dims: [ -1 ] 37 }, 38 { 39 name: "history" 40 data_type: TYPE_STRING 41 dims: [ -1 ] 42 } 43] 44parameters: { //指定python执行环境 45 key: "EXECUTION_ENV_PATH", 46 value: {string_value: "$$TRITON_MODEL_DIRECTORY/../python-3-8.tar.gz"} 47} 48instance_group [ //模型实例组 49 { 50 count: 1 //实例数量 51 kind: KIND_GPU //实例类型 52 gpus: [ 0 ] //指定实例可用的GPU索引 53 } 54] 55
其中必填项为最小模型配置，模型配置文件更多信息可以参考： https://github.com/triton-inference-server/server/blob/r22.04/docs/model_configuration.md
4. 自定义python backend
主要需要实现model.py 中提供的三个接口：
①. initialize: 初始化该Python模型时会进行调用，一般执行获取输出信息及创建模型的操作
②. execute: python模型接收请求时的执行函数；
③. finalize: 删除模型时会进行调用；
如果有 n 个模型实例，那么会调用 n 次initialize 和 finalize这两个函数。
ChatGLM2-6B的model.py文件可以参考如下：
1import os 2# 设置显存空闲block最大分割阈值 3os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'max_split_size_mb:32' 4# 设置work目录 5 6os.environ['TRANSFORMERS_CACHE'] = os.path.dirname(os.path.abspath(__file__))+"/work/" 7os.environ['HF_MODULES_CACHE'] = os.path.dirname(os.path.abspath(__file__))+"/work/" 8 9import json 10 11# triton_python_backend_utils is available in every Triton Python model. You 12# need to use this module to create inference requests and responses. It also 13# contains some utility functions for extracting information from model_config 14# and converting Triton input/output types to numpy types. 15import triton_python_backend_utils as pb_utils 16import sys 17import gc 18import time 19import logging 20import torch 21from transformers import AutoTokenizer, AutoModel 22import numpy as np 23 24gc.collect() 25torch.cuda.empty_cache() 26 27logging.basicConfig(format='%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s', 28 level=logging.INFO) 29 30class TritonPythonModel: 31 """Your Python model must use the same class name. Every Python model 32 that is created must have "TritonPythonModel" as the class name. 33 """ 34 35 def initialize(self, args): 36 """`initialize` is called only once when the model is being loaded. 37 Implementing `initialize` function is optional. This function allows 38 the model to intialize any state associated with this model. 39 40 Parameters 41 ---------- 42 args : dict 43 Both keys and values are strings. The dictionary keys and values are: 44 * model_config: A JSON string containing the model configuration 45 * model_instance_kind: A string containing model instance kind 46 * model_instance_device_id: A string containing model instance device ID 47 * model_repository: Model repository path 48 * model_version: Model version 49 * model_name: Model name 50 """ 51 # You must parse model_config. JSON string is not parsed here 52 self.model_config = json.loads(args['model_config']) 53 54 output_response_config = pb_utils.get_output_config_by_name(self.model_config, "response") 55 output_history_config = pb_utils.get_output_config_by_name(self.model_config, "history") 56 57 # Convert Triton types to numpy types 58 self.output_response_dtype = pb_utils.triton_string_to_numpy(output_response_config['data_type']) 59 self.output_history_dtype = pb_utils.triton_string_to_numpy(output_history_config['data_type']) 60 61 ChatGLM_path = os.path.dirname(os.path.abspath(__file__))+"/ChatGLM2_6B" 62 self.tokenizer = AutoTokenizer.from_pretrained(ChatGLM_path, trust_remote_code=True) 63 model = AutoModel.from_pretrained(ChatGLM_path, 64 torch_dtype=torch.bfloat16, 65 trust_remote_code=True).half().cuda() 66 self.model = model.eval() 67 logging.info("model init success") 68 69 def execute(self, requests): 70 """`execute` MUST be implemented in every Python model. `execute` 71 function receives a list of pb_utils.InferenceRequest as the only 72 argument. This function is called when an inference request is made 73 for this model. Depending on the batching configuration (e.g. Dynamic 74 Batching) used, `requests` may contain multiple requests. Every 75 Python model, must create one pb_utils.InferenceResponse for every 76 pb_utils.InferenceRequest in `requests`. If there is an error, you can 77 set the error argument when creating a pb_utils.InferenceResponse 78 79 Parameters 80 ---------- 81 requests : list 82 A list of pb_utils.InferenceRequest 83 84 Returns 85 ------- 86 list 87 A list of pb_utils.InferenceResponse. The length of this list must 88 be the same as `requests` 89 90 """ 91 output_response_dtype = self.output_response_dtype 92 output_history_dtype = self.output_history_dtype 93 94 # output_dtype = self.output_dtype 95 responses = [] 96 # Every Python backend must iterate over everyone of the requests 97 # and create a pb_utils.InferenceResponse for each of them. 98 for request in requests: 99 prompt = pb_utils.get_input_tensor_by_name(request, "prompt").as_numpy()[0] 100 prompt = prompt.decode('utf-8') 101 history_origin = pb_utils.get_input_tensor_by_name(request, "history").as_numpy() 102 if len(history_origin) > 0: 103 history = np.array([item.decode('utf-8') for item in history_origin]).reshape((-1,2)).tolist() 104 else: 105 history = [] 106 temperature = pb_utils.get_input_tensor_by_name(request, "temperature").as_numpy()[0] 107 temperature = float(temperature.decode('utf-8')) 108 max_token = pb_utils.get_input_tensor_by_name(request, "max_token").as_numpy()[0] 109 max_token = int(max_token.decode('utf-8')) 110 history_len = pb_utils.get_input_tensor_by_name(request, "history_len").as_numpy()[0] 111 history_len = int(history_len.decode('utf-8')) 112 113 # 日志输出传入信息 114 in_log_info = { 115 "in_prompt":prompt, 116 "in_history":history, 117 "in_temperature":temperature, 118 "in_max_token":max_token, 119 "in_history_len":history_len 120 } 121 logging.info(in_log_info) 122 response,history = self.model.chat(self.tokenizer, 123 prompt, 124 history=history[-history_len:] if history_len > 0 else [], 125 max_length=max_token, 126 temperature=temperature) 127 # 日志输出处理后的信息 128 out_log_info = { 129 "out_response":response, 130 "out_history":history 131 } 132 logging.info(out_log_info) 133 response = np.array(response) 134 history = np.array(history) 135 136 response_output_tensor = pb_utils.Tensor("response",response.astype(self.output_response_dtype)) 137 history_output_tensor = pb_utils.Tensor("history",history.astype(self.output_history_dtype)) 138 139 final_inference_response = pb_utils.InferenceResponse(output_tensors=[response_output_tensor,history_output_tensor]) 140 responses.append(final_inference_response) 141 # Create InferenceResponse. You can set an error here in case 142 # there was a problem with handling this inference request. 143 # Below is an example of how you can set errors in inference 144 # response: 145 # 146 # pb_utils.InferenceResponse( 147 # output_tensors=..., TritonError("An error occured")) 148 149 # You should return a list of pb_utils.InferenceResponse. Length 150 # of this list must match the length of `requests` list. 151 return responses 152 153 def finalize(self): 154 """`finalize` is called only once when the model is being unloaded. 155 Implementing `finalize` function is OPTIONAL. This function allows 156 the model to perform any necessary clean ups before exit. 157 """ 158 print('Cleaning up...') 159
5. 部署测试
① 选择9n-triton-devel-gpu-v0.3镜像创建notebook测试实例；
② 把模型放在/9n-triton-devel/model_repository目录下，模型目录结构参考3.1；
③ 进入/9n-triton-devel/server/目录，拉取最新版本的bin并解压：wget http://storage.jd.local/com.bamboo.server.product/7196560/9n_predictor_server.tgz
④ 修改/9n-triton-devel/server/start.sh 为如下：
1mkdir logs 2\rm -rf /9n-triton-devel/server/logs/* 3\rm -rf /tmp/python_env_* 4export LD_LIBRARY_PATH=/9n-triton-devel/server/lib/:$LD_LIBRARY_PATH 5nohup ./bin/9n_predictor_server --flagfile=./conf/server.gflags 2>&1 >/dev/null & 6sleep 2 7pid=`ps x |grep "9n_predictor_server" | grep -v "grep" | grep -v "ldd" | grep -v "stat" | awk '{print $1}'` 8echo $pid 9 10 11
⑤ 运行 /9n-triton-devel/server/start.sh 脚本
⑥ 检查服务启动成功（ChatGLM2-6B模型启动，差不多13分钟左右）
方法1:查看8010端口是否启动：netstat -natp | grep 8010
方法2:查看日志：cat /9n-triton-devel/server/logs/predictor_core.INFO
⑦ 编写python grpc client访问测试服务脚本，放于/9n-triton-devel/client/目录下，访问端口为8010，ip为127.0.0.1，可以参考如下：
1#!/usr/bin/python3 2# -*- coding: utf-8 -*- 3import sys 4sys.path.append('./base') 5from multi_backend_client import MultiBackendClient 6import triton_python_backend_utils as python_backend_utils 7import multi_backend_message_pb2 8 9import time 10import argparse 11import io 12import os 13import numpy as np 14import json 15import struct 16 17def print_result(response, batch_size ): 18 print("outputs len:" + str(len(response.outputs))) 19 20 if (response.error_code == 0): 21 print("response : ", response) 22 23 print(f'res shape: {response.outputs[0].shape}') 24 res = python_backend_utils.deserialize_bytes_tensor(response.raw_output_contents[0]) 25 for i in res: 26 print(i.decode()) 27 28 print(f'history shape: {response.outputs[1].shape}') 29 history = python_backend_utils.deserialize_bytes_tensor(response.raw_output_contents[1]) 30 for i in history: 31 print(i.decode()) 32 33 34def send_one_request(sender, request_pb, batch_size): 35 succ, response = sender.send_req(request_pb) 36 if succ: 37 print_result(response, batch_size) 38 else: 39 print('send_one_request fail ', response) 40 41def send_request(ip, port, temperature, max_token, history_len, batch_size=1, send_cnt=1): 42 request_sender = MultiBackendClient(ip, port) 43 44 request = multi_backend_message_pb2.ModelInferRequest() 45 request.model_name = "chatglm2-6b" 46 47 # 输入占位 48 input0 = multi_backend_message_pb2.ModelInferRequest().InferInputTensor() 49 input0.name = "prompt" 50 input0.datatype = "BYTES" 51 input0.shape.extend([1]) 52 53 input1 = multi_backend_message_pb2.ModelInferRequest().InferInputTensor() 54 input1.name = "history" 55 input1.datatype = "BYTES" 56 input1.shape.extend([-1]) 57 58 input2 = multi_backend_message_pb2.ModelInferRequest().InferInputTensor() 59 input2.name = "temperature" 60 input2.datatype = "BYTES" 61 input2.shape.extend([1]) 62 63 input3 = multi_backend_message_pb2.ModelInferRequest().InferInputTensor() 64 input3.name = "max_token" 65 input3.datatype = "BYTES" 66 input3.shape.extend([1]) 67 68 input4 = multi_backend_message_pb2.ModelInferRequest().InferInputTensor() 69 input4.name = "history_len" 70 input4.datatype = "BYTES" 71 input4.shape.extend([1]) 72 73 query = '请给出一个具体示例' 74 input0.contents.bytes_contents.append(bytes(query, encoding="utf8")) 75 request.inputs.extend([input0]) 76 77 history_origin = np.array([['你知道鸡兔同笼问题么', '鸡兔同笼问题是一个经典的数学问题，涉及到基本的代数方程和解题方法。问题描述为：在一个笼子里面，有若干只鸡和兔子，已知它们的总数和总腿数，问鸡和兔子的数量各是多少？\n\n解法如下：假设鸡的数量为x，兔子的数量为y，则总腿数为2x+4y。根据题意，可以列出方程组：\n\nx + y = 总数\n2x + 4y = 总腿数\n\n通过解方程组，可以求得x和y的值，从而确定鸡和兔子的数量。']]).reshape((-1,)) 78 history = [bytes(item, encoding="utf8") for item in history_origin] 79 input1.contents.bytes_contents.extend(history) 80 request.inputs.extend([input1]) 81 82 input2.contents.bytes_contents.append(bytes(temperature, encoding="utf8")) 83 request.inputs.extend([input2]) 84 85 input3.contents.bytes_contents.append(bytes(max_token, encoding="utf8")) 86 request.inputs.extend([input3]) 87 88 input4.contents.bytes_contents.append(bytes(history_len, encoding="utf8")) 89 request.inputs.extend([input4]) 90 91 # 输出占位 92 output_tensor0 = multi_backend_message_pb2.ModelInferRequest().InferRequestedOutputTensor() 93 output_tensor0.name = "response" 94 request.outputs.extend([output_tensor0]) 95 96 output_tensor1 = multi_backend_message_pb2.ModelInferRequest().InferRequestedOutputTensor() 97 output_tensor1.name = "history" 98 request.outputs.extend([output_tensor1]) 99 100 min_ms = 0 101 max_ms = 0 102 avg_ms = 0 103 for i in range(send_cnt): 104 start = time.time_ns() 105 send_one_request(request_sender, request, batch_size) 106 cost = (time.time_ns()-start)/1000000 107 print ("idx:%d cost ms:%d" % (i, cost)) 108 if cost > max_ms: 109 max_ms = cost 110 if cost < min_ms or min_ms==0: 111 min_ms = cost 112 avg_ms += cost 113 avg_ms /= send_cnt 114 print("cnt=%d max=%dms min=%dms avg=%dms" % (send_cnt, max_ms, min_ms, avg_ms)) 115 116if __name__ == '__main__': 117 parser = argparse.ArgumentParser() 118 parser.add_argument( '-ip', '--ip_address', help = 'ip address', default='127.0.0.1', required=False) 119 parser.add_argument( '-p', '--port', help = 'port', default='8010', required=False) 120 parser.add_argument( '-t', '--temperature', help = 'temperature', default='0.01', required=False) 121 parser.add_argument( '-m', '--max_token', help = 'max_token', default='16000', required=False) 122 parser.add_argument( '-hl', '--history_len', help = 'history_len', default='10', required=False) 123 parser.add_argument( '-b', '--batch_size', help = 'batch size', default=1, required=False, type = int) 124 parser.add_argument( '-c', '--send_count', help = 'send count', default=1, required=False, type = int) 125 args = parser.parse_args() 126 send_request(args.ip_address, args.port, args.temperature, args.max_token, args.history_len, args.batch_size, args.send_count) 127
通用predictor请求格式可以参考： https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/grpc_predict_v2.proto
6. 模型部署
九数算法中台提供了两种部署模型服务方式，分别为界面部署和SDK部署。利用界面中的模型部署只支持JSF协议接口，若要提供JSF服务接口，则可以参考 http://easyalgo.jd.com/help/%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/%E6%A8%A1%E5%9E%8B%E8%AE%A1%E7%AE%97/%E6%A8%A1%E5%9E%8B%E9%83%A8%E7%BD%B2.html 直接部署。
由于我后续需要将ChatGLM2-6B模型集成至langchain中使用，所以对外提供http协议接口比较便利，经与算法中台同学请教后使用SDK方式部署可以满足。由于界面部署和SDK部署目前研发没有对齐，用界面部署时直接可以使用3.1中的模型结构，使用SDK部署则需要调整模型结构如下：
同时需要在config.pbtxt中将执行环境路径设置如下：
1parameters: { 2 key: "EXECUTION_ENV_PATH", 3 value: {string_value: "$$TRITON_MODEL_DIRECTORY/1/python-3-8.tar.gz"} 4} 5
模型部署代码可以参考如下：
1from das.triton.model import TritonModel 2 3model = TritonModel("chatglm2-6b") 4 5predictor = model.deploy( 6 path="$pwd/model_repository/chatglm2-6b", # 模型文件所在的目录 7 protocol='http', 8 endpoint = "9n-das-serving-lf2.jd.local", 9 cpu=4, 10 memory=30, 11 use_gpu=True, # 根据是否需要gpu加速推理来配置 12 override = True, 13 instances=2 14 ) 15
四.集成至langchain
使用langchain可以快速基于LLM模型开发一些应用。使用LLMs模块封装ChatGLM2-6B，请求我们的模型服务，主要实现_call函数，可以参考如下代码：
1 2import json 3import time 4import base64 5import struct 6import requests 7import numpy as np 8from pathlib import Path 9from abc import ABC, abstractmethod 10from langchain.llms.base import LLM 11from langchain.llms import OpenAI 12from langchain.llms.utils import enforce_stop_tokens 13from typing import Dict, List, Optional, Tuple, Union, Mapping, Any 14 15import warnings 16warnings.filterwarnings("ignore") 17 18class ChatGLM(LLM): 19 max_token = 32000 20 temperature = 0.01 21 history_len = 10 22 url = "" 23 def __init__(self): 24 super(ChatGLM, self).__init__() 25 26 @property 27 def _llm_type(self): 28 return "ChatGLM2-6B" 29 30 @property 31 def _history_len(self) -> int: 32 return self.history_len 33 34 @property 35 def _max_token(self) -> int: 36 return self.max_token 37 38 @property 39 def _temperature(self) -> float: 40 return self.temperature 41 42 def _deserialize_bytes_tensor(self, encoded_tensor): 43 """ 44 Deserializes an encoded bytes tensor into an 45 numpy array of dtype of python objects 46 Parameters 47 ---------- 48 encoded_tensor : bytes 49 The encoded bytes tensor where each element 50 has its length in first 4 bytes followed by 51 the content 52 Returns 53 ------- 54 string_tensor : np.array 55 The 1-D numpy array of type object containing the 56 deserialized bytes in 'C' order. 57 """ 58 strs = list() 59 offset = 0 60 val_buf = encoded_tensor 61 while offset < len(val_buf): 62 l = struct.unpack_from("<I", val_buf, offset)[0] 63 offset += 4 64 sb = struct.unpack_from("<{}s".format(l), val_buf, offset)[0] 65 offset += l 66 strs.append(sb) 67 return (np.array(strs, dtype=np.object_)) 68 69 @classmethod 70 def _infer(cls, url, query, history, temperature, max_token, history_len): 71 query = base64.b64encode(query.encode('utf-8')).decode('utf-8') 72 history_origin = np.asarray(history).reshape((-1,)) 73 history = [base64.b64encode(item.encode('utf-8')).decode('utf-8') for item in history_origin] 74 temperature = base64.b64encode(temperature.encode('utf-8')).decode('utf-8') 75 max_token = base64.b64encode(max_token.encode('utf-8')).decode('utf-8') 76 history_len = base64.b64encode(history_len.encode('utf-8')).decode('utf-8') 77 data = { 78 "model_name": "chatglm2-6b", 79 "inputs": [ 80 {"name": "prompt", "datatype": "BYTES", "shape": [1], "contents": {"bytes_contents": [query]}}, 81 {"name": "history", "datatype": "BYTES", "shape": [-1], "contents": {"bytes_contents": history}}, 82 {"name": "temperature", "datatype": "BYTES", "shape": [1], "contents": {"bytes_contents": [temperature]}}, 83 {"name": "max_token", "datatype": "BYTES", "shape": [1], "contents": {"bytes_contents": [max_token]}}, 84 {"name": "history_len", "datatype": "BYTES", "shape": [1], "contents": {"bytes_contents": [history_len]}} 85 ], 86 "outputs": [{"name": "response"}, 87 {"name": "history"}] 88 } 89 response = requests.post(url = url, 90 data = json.dumps(data, ensure_ascii=True), 91 headers = {"Content_Type": "application/json"}, 92 timeout=120) 93 return response 94 95 def _call(self, 96 query: str, 97 history: List[List[str]] =[], 98 stop: Optional[List[str]] =None): 99 temperature = str(self.temperature) 100 max_token = str(self.max_token) 101 history_len = str(self.history_len) 102 url = self.url 103 response = self._infer(url, query, history, temperature, max_token, history_len) 104 if response.status_code!=200: 105 return "查询结果错误" 106 if stop is not None: 107 response = enforce_stop_tokens(response, stop) 108 result = json.loads(response.text) 109 # 处理response 110 res = base64.b64decode(result['raw_output_contents'][0].encode('utf-8')) 111 res_response = self._deserialize_bytes_tensor(res)[0].decode() 112 return res_response 113 114 def chat(self, 115 query: str, 116 history: List[List[str]] =[], 117 stop: Optional[List[str]] =None): 118 temperature = str(self.temperature) 119 max_token = str(self.max_token) 120 history_len = str(self.history_len) 121 url = self.url 122 response = self._infer(url, query, history, temperature, max_token, history_len) 123 if response.status_code!=200: 124 return "查询结果错误" 125 if stop is not None: 126 response = enforce_stop_tokens(response, stop) 127 result = json.loads(response.text) 128 # 处理response 129 res = base64.b64decode(result['raw_output_contents'][0].encode('utf-8')) 130 res_response = self._deserialize_bytes_tensor(res)[0].decode() 131 # 处理history 132 history_shape = result['outputs'][1]["shape"] 133 history_enc = base64.b64decode(result['raw_output_contents'][1].encode('utf-8')) 134 res_history = np.array([i.decode() for i in self._deserialize_bytes_tensor(history_enc)]).reshape(history_shape).tolist() 135 return res_response, res_history 136 137 @property 138 def _identifying_params(self) -> Mapping[str, Any]: 139 """Get the identifying parameters. 140 """ 141 _param_dict = { 142 "url": self.url 143 } 144 return _param_dict 145
注意：模型服务调用url等于在模型部署页面调用信息URL后加上" MutilBackendService/Predict "
五.总结
本文详细介绍了在集团9n-triton工具上部署ChatGLM2-6B过程，希望可以为有部署需求的同学提供一些帮助。
作者：京东保险 赵风龙
来源：京东云开发者社区 转载请注明出处