RECENT POSTS

Do you have GDPR compliance issues ?

Check out Legiscope a GDPR compliance software, that will save you weeks of work, automating your documentation, the training of your teams and all processes you need to keep your organisation compliant with privacy regulations

P5-lingua-zh-wordsegmenter

Jul 20, 2023

Simplified Chinese Word Segmentation

This is a perl version of simplified Chinese word segmentation.

The algorithm for this segmenter is to search the longest word at each point from both left and right directions, and choose the one with higher frequency product.

The original program is from the CPAN module LinguaZHWordSegment https//metacpan.org/author/CHENYR I did the follwing changes 1 make the interface object oriented; 2 make the internal string into utf8; 3 using sogou’s dictionary http//www.sogou.com/labs/dl/w.html as the default dictionary.

Checkout these related ports: