响应式网站怎么做mip,个人网站的制作教程,手机搭建网站软件下载,重庆建设部网站1.学习内容
本节次学习内容来自于吴恩达老师的Preprocessing Unstructured Data for LLM Applications课程#xff0c;因涉及到非结构化数据的相关处理#xff0c;遂做学习整理。 本节主要学习pdf中的表格数据处理
2.环境准备
和之前一样#xff0c;可以参考LLM应用构建前…1.学习内容
本节次学习内容来自于吴恩达老师的Preprocessing Unstructured Data for LLM Applications课程因涉及到非结构化数据的相关处理遂做学习整理。 本节主要学习pdf中的表格数据处理
2.环境准备
和之前一样可以参考LLM应用构建前的非结构化数据处理一标准化处理认识数据 其中配置信息保持一致
同样的需要unstructured.io上获取APIkey。
3.开始尝试
3.1导入环境
# Warning control
import warnings
warnings.filterwarnings(ignore)from unstructured_client import UnstructuredClient
from unstructured_client.models import shared
from unstructured_client.models.errors import SDKErrorfrom unstructured.staging.base import dict_to_elements# 初始化API
s UnstructuredClient(api_key_authXXX,server_urlhttps://api.unstrXXX,
)3.2样例浏览
from IPython.display import Image
Image(filenameimages/embedded-images-tables.jpg, height600, width600) 输出如下
3.3处理pdf文档
filename example_files/embedded-images-tables.pdfwith open(filename, rb) as f:filesshared.Files(contentf.read(),file_namefilename,)req shared.PartitionParameters(filesfiles,strategyhi_res,hi_res_model_nameyolox,skip_infer_table_types[],pdf_infer_table_structureTrue,
)try:resp s.general.partition(req)elements dict_to_elements(resp.elements)
except SDKError as e:print(e)# 找到处理数据中的Table元素的unstructured对象数据
tables [el for el in elements if el.category Table]
tables[0].text输出如下
Inhibitor Polarization Corrosion be (V/dec) ba (V/dec) Ecorr (V) icorr (AJcm?) concentration (g) resistance (Q) rate (mmj/year) 0.0335 0.0409 —0.9393 0.0003 24.0910 2.8163 1.9460 0.0596 .8276 0.0002 121.440 1.5054 0.0163 0.2369 .8825 0.0001 42121 0.9476 s NO 03233 0.0540 —0.8027 5.39E-05 373.180 0.4318 0.1240 0.0556 .5896 5.46E-05 305.650 0.3772 5 0.0382 0.0086 .5356 1.24E-05 246.080 0.0919将其转为html形式
table_html tables[0].metadata.text_as_html
table_html输出如下
tabletheadtrthInhibitor concentration (g)/ththbe (V/dec)/ththba (V/dec)/ththEcorr (V)/ththicorr (AJcm?)/ththPolarization resistance (Q)/ththCorrosion rate (mmj/year)/th/tr/theadtbodytrtd/tdtd0.0335/tdtd0.0409/tdtd—0.9393/tdtd0.0003/tdtd24.0910/tdtd2.8163/td/trtrtdNO/tdtd1.9460/tdtd0.0596/tdtd—0.8276/tdtd0.0002/tdtd121.440/tdtd1.5054/td/trtrtd/tdtd0.0163/tdtd0.2369/tdtd—0.8825/tdtd0.0001/tdtd42121/tdtd0.9476/td/trtrtds/tdtd03233/tdtd0.0540/tdtd—0.8027/tdtd5.39E-05/tdtd373.180/tdtd0.4318/td/trtrtd/tdtd0.1240/tdtd0.0556/tdtd—0.5896/tdtd5.46E-05/tdtd305.650/tdtd0.3772/td/trtrtd 5/tdtd0.0382/tdtd0.0086/tdtd—0.5356/tdtd1.24E-05/tdtd246.080/tdtd0.0919/td/tr/tbody/table3.4 格式化呈现
from io import StringIO
from lxml import etreeparser etree.XMLParser(remove_blank_textTrue)
file_obj StringIO(table_html)
tree etree.parse(file_obj, parser)
print(etree.tostring(tree, pretty_printTrue).decode())输出如下
tabletheadtrthInhibitor concentration (g)/ththbe (V/dec)/ththba (V/dec)/ththEcorr (V)/ththicorr (AJcm?)/ththPolarization resistance (Q)/ththCorrosion rate (mmj/year)/th/tr/theadtbodytrtd/td0.0335/tdtd0.0409/tdtd#8212;0.9393/tdtd0.0003/tdtd24.0910/tdtd2.8163/td/trtrtdNO/tdtd1.9460/tdtd0.0596/tdtd#8212;0.8276/tdtd0.0002/tdtd121.440/tdtd1.5054/td/trtrtd/td0.0163/tdtd0.2369/tdtd#8212;0.8825/tdtd0.0001/tdtd42121/tdtd0.9476/td/trtrtds/tdtd03233/tdtd0.0540/tdtd#8212;0.8027/tdtd5.39E-05/tdtd373.180/tdtd0.4318/td/trtrtd/td0.1240/tdtd0.0556/tdtd#8212;0.5896/tdtd5.46E-05/tdtd305.650/tdtd0.3772/td/trtrtd 5/tdtd0.0382/tdtd0.0086/tdtd#8212;0.5356/tdtd1.24E-05/tdtd246.080/tdtd0.0919/td/tr/tbody
/table
3.5 还原表格到html中显示
from IPython.core.display import HTML
HTML(table_html)输出如下
3.6 借助langchain进行摘要
from langchain_openai import ChatOpenAI
from langchain_core.documents import Document
from langchain.chains.summarize import load_summarize_chainllm ChatOpenAI(temperature0, model_namegpt-3.5-turbo-1106)
chain load_summarize_chain(llm, chain_typestuff)
chain.invoke([Document(page_contenttable_html)])输出如下
{input_documents: [Document(page_contenttabletheadtrthInhibitor concentration (g)/ththbe (V/dec)/ththba (V/dec)/ththEcorr (V)/ththicorr (AJcm?)/ththPolarization resistance (Q)/ththCorrosion rate (mmj/year)/th/tr/theadtbodytrtd/tdtd0.0335/tdtd0.0409/tdtd—0.9393/tdtd0.0003/tdtd24.0910/tdtd2.8163/td/trtrtdNO/tdtd1.9460/tdtd0.0596/tdtd—0.8276/tdtd0.0002/tdtd121.440/tdtd1.5054/td/trtrtd/tdtd0.0163/tdtd0.2369/tdtd—0.8825/tdtd0.0001/tdtd42121/tdtd0.9476/td/trtrtds/tdtd03233/tdtd0.0540/tdtd—0.8027/tdtd5.39E-05/tdtd373.180/tdtd0.4318/td/trtrtd/tdtd0.1240/tdtd0.0556/tdtd—0.5896/tdtd5.46E-05/tdtd305.650/tdtd0.3772/td/trtrtd 5/tdtd0.0382/tdtd0.0086/tdtd—0.5356/tdtd1.24E-05/tdtd246.080/tdtd0.0919/td/tr/tbody/table)],output_text: The table provides data on the corrosion rate and polarization resistance of different inhibitor concentrations in a solution. The data includes the inhibitor concentration, be and ba values, Ecorr, icorr, polarization resistance, and corrosion rate. The table shows the impact of different inhibitor concentrations on the corrosion rate and polarization resistance.}4. 总结
可以看到非结构化数据识别还是有难度不知道为什么实验中部分识别结果是错的如果追求准确性还是得斟酌一下。
