qurator-spk
/
neat
mirror of https://github.com/qurator-spk/neat.git
1
0
Fork 0
Code Issues Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '9f2f3ba2d5'
${ noResults }
neat/docs/Preprocessing.md

1.6 KiB
Raw Blame History

Preprocessing

The preprocessing pipeline that is developed at the Berlin State Library comprises the following steps:

Layout Analysis & Textline Extraction

Layout Analysis & Textline Extraction @sbb_pixelwise_segmentation

OCR & Word Segmentation

OCR is based on OCR-D's ocrd_tesserocr which requires Tesseract >= 4.1.0. The GT4HistOCR_2000000 model, which is trained on the GT4HistOCR corpus, is used. Further details are available in the paper.

Tokenization

Named Entity Recognition

For Named Entity Recognition, a BERT-Base model was trained for noisy OCR texts with historical spelling variation. sbb_ner is using a combination of unsupervised training on a large (~2.3m pages) corpus of German OCR in combination with supervised training on a small (47k tokens) annotated corpus. Further details are available in the paper.