FreeBSD.software
Home/textproc/py311-tiktoken

py311-tiktoken

0.12.0_2textproc

Fast BPE tokeniser for use with OpenAI's models

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

$pkg install py311-tiktoken
github.com/openai/tiktoken ↗
Origin
textproc/py-tiktoken
Size
2.70MiB
License
MIT
Maintainer
sunpoet@FreeBSD.org
Dependencies
3 packages
Required by
7 packages

Dependencies (3)

python311py311-requestspy311-regex

Required By (7)

chatbladepy311-aider_chatpy311-ddtracepy311-langchain-openaipy311-litellmpy311-openai-whisperpy311-transformers