qurator-spk
/
neat
mirror of https://github.com/qurator-spk/neat.git
1
0
Fork 0
Code Issues Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '9dff8a78ba'
${ noResults }
neat/docs/Preprocessing.md

1.7 KiB
Raw Blame History

Preprocessing

The preprocessing pipeline that is developed at the Berlin State Library comprises the following steps:

Layout Analysis & Textline Extraction

Layout Analysis & Textline Extraction @sbb_pixelwise_segmentation

OCR & Word Segmentation

OCR is based on OCR-D's ocrd_tesserocr which requires Tesseract >= 4.1.0. The Fraktur_5000000 model, which is trained on GT4HistOCR is used. corpiu The PAGE-XML produced by the Layout Analysis & Textline Extraction is taken as input, and the output is PAGE-XML containing the text recognition results with absolute pixel coordinates describing bounding boxes for words.

Tokenization

Named Entity Recognition

For Named Entity Recognition, a BERT-Base model was trained. sbb_ner is using a combination of unsupervised training on a large (~2.3m pages) OCR corpus in combination with supervised training on a small (50k tokens) annotated corpus. Further details are available in the paper.