py311-Mathics_Scanner

Tokeniser, char tables, & conversion routines for the Wolfram Language

This is a tokeniser or scanner portion for the Wolfram Language. As such, it also contains a full set of translation between Wolfram Language named characters, their Unicode/ASCII equivalents and code-points. This is used as the scanner inside Mathics but it can also be used for tokenizing and formatting Wolfram Language code. This library is also quite usefull if you need to work with Wolfram Language named character and convert them to various formats. See also: https://github.com/Mathics3/mathics-scanner