- add CLI options `--num-jobs` and `--halt-fail`
- separate `Eynollah_ocr.run_single()` to be scheduled
- use ProcessPoolExecutor w/ forking and QueueListener
- also skip if input XML file is missing
- log processing times per job and overall
- training.models.CTCDecoder: prefer beam search over greedy
(because it is more accurate)
- training reload makefile: skip ONNX and TF-Serving conversion
for cnn-rnn-ocr models (because these would not work)
- training reload makefile: default to onnx and tf conversions
for all models (tf for training and onnx for inference)
instead of tf-serving export
- `training.models.CTCDecoder`: switch back
from `tf.nn.ctc_beam_search_decoder()`
to `tf.nn.ctc_greedy_decoder()`
(because ONNX only implements `CTCGreedyDecoder`)
- `training.models.cnn_rnn_ocr_model(inference=True)` and
`training.models.cnn_rnn_ocr_model4inference`:
drop layer `tf.io.decode_raw()`
(because ONNX does not implement `DecodePaddedRaw`)
- `Eynollah_ocr.run_cnn()`: expect bytes arrays from predictor
instead of uint8
- `predictor`: to prevent segfaults when sending `tf.string` results
via `shared_memory`, convert `np.object` to `np.bytes_` directly
when rebuilding the inference model for cnn-rnn-ocr,
- open the old `characters_org.txt` file for the charset
- use it to pass the actual `n_classes` (overriding the config)
- use its path to pass the `characters_txt_file` (overriding the config)
Because transformers v4 and v5 API for image preprocessor differs,
and the model-internal image input sizes are actually irrelevant,
because the preprocessor will resize them anyway, and there is no
batch dimension (because the input images will have different shapes),
do not advertise this information in `.input_shape`.
- ModelZoo: drop `trocr_processor` model type
- `ModelZoo.load_models()`: use Predictor for `ocr_tr` models, too
- `ModelZoo.load_model()`: for `ocr_tr`, load processor and model,
then define a function object as stand-in for the common model
interface based on Keras (w/ `.predict_on_batch()`)
- Predictor: allow multi-input without actual batch dimension
for `ocr_tr` models (because the model takes a list of original
image arrays and resizes them to model shape internally)
- Eynollah_ocr: adapt (replacing preprocessing, prediction and
decoding steps by a single `.predict()` call)
- `min_conf_value_of_textline_text`: apply by skipping
lines below threshold (instead of writing empty text),
and delete their TextEquiv (if existing)
- `write_ocr()`: simplify, and ensure consistency between
line and region level text correctly
- drop `end_character` mechanics and `characters` model type
for decoding output probability (not needed)
- drop `decode_batch_predictions()` and `num_to_char` model type
(part of inference model)
- drop roughshot confidence estimation calculation
(returned precisely by inference model)
- adapt model prediction to inference model: just omit zeros,
map to bytes, filter OOV tokens and decode UTF-8 to str
- if no binarization input was provided, then compute it on the fly
using `binarization` model
- also apply `min_conf_value_of_textline_text` (as for TrOCR)
- batching over entire page instead of region-wise
(which underfilled batches)
- simplify and avoid copied redundant code
- rename `extracted_conf_value_merged` → `extracted_confs_merged`
- move `batched()` from `utils.utils_ocr` to `utils`
- drop `utils_ocr.distortion_free_resize()` (not needed)
- simplify `utils_ocr.break_curved_line_into_small_pieces_and_then_merge()`
- drop `utils_ocr.return_textline_contour_with_added_box_coordinate()`
and `utils_ocr.return_rnn_cnn_ocr_of_given_textlines()` (not needed)
- ModelZoo: drop `num_to_char` and `characters` model types,
also drop `_load_characters()` and `_load_num_to_char()` loaders
- `ModelZoo.load_models()`: use Predictor for `ocr` models, too
- `ModelZoo.load_model()`: delegate runtime/inference conversion of
OCR models to `eynollah.training.models.cnn_rnn_ocr_model4inference`
- `training.models`: add (purely functional) Keras layer `CTCDecoder`
for inference on top of softmax output, but using TF backend
function instead of (broken) `Keras.backend.ctc_decode()`, while
switching to beam search (instead of greedy) and also returning
decoded path probability
- `training.models.cnn_rnn_ocr_model()` w/ `inference=True`:
* add kwarg `characters_txt_file` for file path of character set
* configure secondary tensor path on OCR graph for binarized input
(additional input `image_bin`, averaging softmax outputs)
* use new `CTCDecoder` layer and inverse `StringLookup` layer to
decode from softmax output to tf.string; so inference models
now have 2 inputs (RGB, binarized) and 2 outputs (text, prob)
* since `np.dtype=object` cannot be handled by SharedMemory (as
needed by Predictor queues), also replace tf.string by tf.uint8
arrays
* use this for `training convert` for OCR models w/ `--rebuild`
- `training.models.cnn_rnn_ocr_model4inference`:
* new function which does the same but loads an existing OCR model
in training configuration (i.e. without prior `inference=True`)
* use this for `training convert` for OCR models w/o `--rebuild`
- comment out ad-hoc conversion/loading of autosized models
- refactor predictor backends for model types into separate functions
- only attempt inference conversion of cnn-rnn-ocr model
if applicable (`ctc_loss` layer still present)
- apply VRAM limits across model types
(Keras, TF-Serving, ONNX)
- apply TF device selection across model types
(Keras, TF-Serving)
- implement predictor backend for ONNX models:
- using onnxruntime
- covering CUDA and TensorRT providers
- trying to support manual device selection
- hiding session management details
- converting float32 to float16
- move `train_cli` from cli.py to train.py,
add docstring
- add `convert_cli`:
- load any (supported) model format
(i.e. not exported TF-Serving or ONNX)
- if SavedModel format with `config.json` present,
and `--rebuild` is requested, create new model
from `models.get_model()` for this configuration,
and load weights
- if model type is `cnn-rnn-ocr` and configuration
is still for training (`ctc_loss`), then extract
inference model
- apply requested `--format` conversion:
HDF5, Keras native, Keras SavedModel, TF-Serving SavedModel
or ONNX
- if output format is directory (i.e. SavedModel),
then copy over `config.json`, too
- reload-models-v0.8.mk:
- adapt recipe for converter CLI (i.e. `--format tf-serving`
w/ `--rebuild` if possible)
- add targets for other useful data formats
- extend list of model names to all current models
(as all benefit from TF-Serving export)
- cancel ONNX conversion for vision transformer models
(as these do not work, yet)
- drop ad-hoc configuration parameter `reload_weights`
(used for conversion/export of models for inference,
to be replaced by extra CLI)
- re-interprete `dir_of_start_model` to also load weights
if not `continue_training`
- models: add new `get_model()`, passing in Sacred config
to capture builder function arguments
- train: fewer imports
- train: no need to pass `custom_objects` if loading with
`compile=False` (and we custom-compile later, anyway)
- growth strategy is more flexible, but uses much more VRAM
- limit strategy needs to be calibrated to models (currently fixed),
and batch size, but needs much less VRAM and is faster
- re-use Eynollah base class
- use `ModelZoo.load_models()` instead of `load_model()`
- pass in `device` init kwarg, delegate to `ModelZoo.load_models()`
- `device`: return Torch device at loaded model tensors
instead of ad-hoc selection
- make numeric init kwargs non-optional (only numeric)
- `load_models()`: uniformly handle arg types
- `load_model()`: move handling of non-model categories
to `load_models()`
- `load_model()`: move SavedModel preference over HDF5 to `model_path()`
- `_load_ocr_model()`: add user-selected device handling and reporting
for Torch (as for TF)
- `_load_ocr_model()`: move (TF-based) CNN-RNN case to `load_model()`
(including Keras layer mapping)
- `shutdown()`: only apply `shutdown()` to Predictor model types
- found positive and negative peaks, and even more so their
relative offsets, may overflow in the cropped image,
causing fake textlines; avoid that by clipping to the valid
y coordinates
- calculation for number of tiles: sometimes one less
tile is needed by making the previous last tile
half-full on the right side
- add some (commented) plotting
- simplify (a lot, but only partially)
- relative images now need larger relative min_area
(i.e. compensation factors)
- do not attempt (even) single-line skew estimation
(via linear regression) if there is no (large enough)
contour at all
- avoid re-computing `mask_parent`
- add some (commented) plotting
- re-use Eynollah base class, drop copied code
- simplify `run()` and `run_single()`
- delegate to `do_prediction()`
instead of custom (old) tiling loop
- drop `predict()`
- add `--device` option to CLI as well
- re-use Eynollah base class, drop copied code
- write usable `run()` and `run_single()`
- delegate to `resize_image_with_column_classifier()`
for column classifier, resizing and enhancement,
instead of `resize_and_enhance_image_with_column_classifier()`
(which does _not_ actually enhance)
- drop unused `predict_enhancement()`
- add defaults to `num_col` options (always numeric)
- add `--device` option to CLI as well
use rules from `resize_and_enhance_image_with_column_classifier()`
and apply them to `resize_image_with_column_classifier()` as well
(to be used by enhancer CLI)
instead of hard cut-offs between overlapping window tiles,
apply sigmoid attenuation to slide from one to the next
(apply all postprocessing in the end)
- calculation for number of tiles: sometimes one less
tile is needed by making the previous last tile
half-full on the right side
- calculation of window margins: fix case if dimension
extends to full image shape
- simplify (identifiers, slicing etc)
in `estimate_skew_contours()`, distinguish between angle stats
scattering around <45° vs >45°: in the latter case, use modulo
180° for averages - to avoid cancelling out +90° with -90°
- introduce `config_params` key `reload_weights`
- add respective section for all model types:
- build fresh model from code
- load existing weights from `dir_of_start_model`
- save to `dir_output` under same basename as existing model
(but without optimizer and metrics; which does not work currently)
- exit immediately (i.e. no actual training)
- reorder so reload_weights is after compilation but before data loading
- move `extract_page()` to the start (right after enhancement),
so early layout and textline model prediction sees cropped
image
- `extract_page()`: also return page mask
- `get_early_layout()`:
* use cropped image
* also run optional table prediction here,
map table label and confidence already
(so no need to pass these arrays everywhere)
* suppress all non-text type regions in textline mask
* also return text+table mask
(so no need to reconstruct it everywhere)
- apply page mask to textline mask and early layout result
(i.e. suppress areas beyond border contour)
- `run_graphics_and_columns()`:
* rename → `run_columns()`
* no table prediction here
* no page extraction here
* no page cropping+masking here
* no textline mask suppression here
- `run_graphics_and_columns_without_layout()`: drop
(not needed anymore)
- `run_marginals()` vs. `get_marginals()`: extract
`text_mask` internally from early layout
- early page cropping for col-classifier:
also use cropped image in input binarization mode
- early page cropping for col-classifier:
get external contours instead of indiscriminate tree
- writer: skip layout mode now also uses cropped coordinates
(so drop kwarg for it)
for local (within-box) ordering of region contours, use the same
text mask (merely eroded) as for the contour extraction itself:
the text+table+drop mask from early+full layout prediction,
rather than the textline mask, because the latter may be empty
in some boxes and is unlikely to be more useful than the region
mask itself
(as there are valid cases where both left and right marginalia
is present) follow-up 4bdea39 by re-allowing left point _and_
right point - but still score-based, and not if very asymmetric
- `get_marginals` modifies region labels in-place anyways,
so no need for retval
- de/rotate only inside `get_marginals` (for consistency)
- return early if no marginals detected
- `run_marginals`: only useful in 1 or 2 columns, so keep to
that conditional branch; allows avoiding unnecessary resizing
of images to and fro
- rename `text_regions_p_1` → `text_regions_p`
in search of valid peaks (gaps between text columns),
- drop absolute values for minimum gap depth
(likely crafted for some fixed resolution examples)
- instead, use criterion relative to maximum column depth
and page height (trying to loosely approximate the prior
constants, albeit somewhat more permissive)
in search of valid (above threshold) peaks:
- do not just pick right-most left and left-most right span;
- instead,
* if no peaks on the left, then only search right
* if no peaks on the right, then only search left
* if peaks on both sides, then only better side
(so never return marginals on both sides!)
* use scoring for peaks that reflects their peak
prominence and peak height (but keep positional
range constraints for what constitues left and right)
- rename `thickness_along_y_percent` →
`max_textline_thickness_percent`
- rename `marginlas_should_be_main_text` →
`main_text_should_be_marginals`
- constrain `find_peaks()` by prominence and distance
- simplify (a lot)
- add comments for possible improvements
and for plotting
- use new `rotate_image_enlarge` instead of
custom (insufficient) padding w/ `rotate_image`
- get external contours instead of tree
(without checking hierarchy afterwards)
- use largest textline contours by area instead of
longest polygon path
- always use `separate_lines` (but without its incorrect
angle/offset calculations) instead of `separate_lines_vertical_cont`
- calculate coordinate transformation (shift, angle)
for all cases (including >45°)
- simplify
- use relative images, cropped to parent bbox (faster)
- no `scale` parameter (unused)
- use largest textline contours by area instead of first
- simplify
- return early if textline mask is empty
- intersect textline mask with parent mask
(so neighbouring, truncated textlines
will not interfere)
- fix bug when resulting angle is small:
rather, compare with page angle
- if there is more than 1 line in the region,
* use median instead of mean to estimate y_diff
* if height dominates over width and x_diff
over y_diff, then assume 90°: transpose image,
deskew on that, then add 90° to result
- otherwise instead of just using page angle,
try to estimate single-line angle by approximating
slope of linear x-y regression on mask image;
again, if height dominates over width, then
assume +90° and use transposed image
- drop unused `scale` param
- when merging large line with small lines,
don't use first new contour but largest
- get external contours instead of tree
(without checking hierarchy afterwards)
- simplify
- rename `get_regions()` → `get_early_layout()`
- split up `run_boxes_no/full_layout()` into shared
* `get_full_layout()` (for lapping mapping,
table decoding and optional full model prediction)
* `get_deskewed_masks()` (for de-rotation)
* extraction of various region types (polygons and confidences)
* `run_boxes_order()` (for column detection and box ordering)
- rename `contours_tables` → `polygons_of_tables`
This further reduces redundant code, avoids splitting up the same
functionality across different places depending on mode etc.
- `run_single`: re-use `return_contours_of_interested_region`
for extraction and filtering of text region contours
- `run_single`: isolate new function `match_deskewed_contours`
- `run_single`: apply dilation afterwards
- rename `contours_only_text_parent_d_ordered` → `polygons_of_textregions_d`
- rename `contours_only_text_parent` → `polygons_of_textregions`
- rename `contours_only_text_parent_h` → `polygons_of_textregions_h`
- `do_work_of_slopes_new_curved` and `get_slopes_and_deskew_new_curved`:
no need for `mask_texts_only` array arg
- `filter_contours_inside_a_bigger_one`: no need for `image` as array arg,
simplify
- `split_textregion_main_vs_head`: simplify, re-order arguments
and return tuple logically
- if no main text regions are found, just convert marginals to main text
and continue normally instead of stopping early w/ empty marginals (i.e.
no textlines)
- do_order_of_regions_with_model:
* add `polygons_of_drop_capitals`, order these indices as well
(model was not trained for this, but it works)
* explicit label identifiers instead of number literals
* map marginals and images correctly
* simplify (a lot)
* reduce inference batch size to accomodate 8 GB VRAM GPUs
- return_indexes_of_contours_located_inside_another_list_of_contours:
simplify
- pass on probabilities from predicted class everywhere
- rename `confidence_matrix` → `confidence_regions` / `regions_confidence`
- rename `get_textregion_confidences()` → `get_region_confidences()`
- add same for tables, textlines and regionsfl (full layout model)
- aggregate per-region confidence lists for image, table, drop-capital,
left marginal and right marginal regions
- add in writer
- simplify/re-indent some
- try to replace more number literals with class label identifiers
- re-introduce boosting `heading` thresholding broken
when refactoring (light version and do_prediction)
- also return confidence for full layout prediction
1. use connected component analysis to get unique segments
in early prediction result
2. for each drop-capital segment in full prediction result,
find matching early segment
3. when they have high overlap, assign drop-capital label
to the entire early segment
- rename `putt_bb_of_drop_capitals_of_model_in_patches_in_layout`
→ `fill_bb_of_drop_capitals`
- also allow image (besides text) label in early layout prediction
result when checking if entire bbox can be filled (as opposed to
just drop-capital | image | background mask)
- simplify
fix bug where in non-full mode, the wrong class label was assumed
for separator regions (3 in non- vs 6 in full layout mode):
- pass in separator mask instead of full segmentation map
- rename for clarity:
- `regions_without_separators` → `text_mask` (alread binary)
- `regions_with_separators` → `sep_mask` (now just binary)
(thresholding and decoding with artificial boundary class can
overwrite existing column separators, which in turn can contribute
to missing column boundaries; this prioritises seps over boundaries,
which does not impair separation of instances, as seps will separate
text/image/etc instances just as well as artificial boundaries)
When 338c4a0e wrapped all prediction models for automatic
image size adaptation in CUDA,
- tiling (`_patched`) was indeed faster
- whole (`_resized`) was actually slower
But CUDA-based tiling also increases GPU memory requirements
a lot. And with the new parallel subprocess predictors, Numpy-
based tiling is not necessarily slower anymore.
(avoid strange image handling short-cut, which uses
early cropped image used for column classification
instead of normal image in 1/2-column cases;
fixes accuracy issues of region_1_2 model on these images)
(as follow-up to ec08004f:)
- create log queues and QueueListener separately for each job
- receive job logs sequentially
- drop log filter mechanism (prefixing log messages by file name)
- also count ratio of successful jobs
allow setting device specifier to load models into
either
- CPU or single GPU0, GPU1 etc
- per-model patterns, e.g. col*:CPU,page:GPU0,*:GPU1
pass through as kwargs until `ModelZoo.load_models()` setup up TF
- Eynollah: instead of one `Predictor` instance as stand-in for
entire `ModelZoo`, keep the latter but have each model in `_loaded`
dict become an independent predictor instance
- `ModelZoo.load_models()`: instantiate `Predictor`s for each
`model_category` and then call `Predictor.load_model()` on them
- `Predictor.load_model()`: set args/kwargs for `ModelZoo.load_model()`,
then spawn subprocess via `.start()`, which first enters `setup()`...
- `Predictor.setup()`: call `ModelZoo.load_model()` instead of (plural)
`.load_models()`; save to `self.model` instead of `self.model_zoo`
- `ModelZoo.load_model()`: move _all_ CUDA configuration and
TF/Keras-specific module initialization here (to be used only by
predictor subprocess)
- `Predictor`: drop stand-in `SingleModelPredictor` retrieved by `get()`;
directly provide `predict()` and `output_shape` via `self.call()`
- `Predictor`: drop `model` arg from all queues - now implicit; use
`self.name` for model name in messages
- `Predictor`: no need for requeuing other tasks (only same model now)
- `Predictor`: reduce rebatching batch sizes due to increased VRAM footprint
- `Eynollah.setup_models()`: set up loading `_patched` / `_resized`
here instead of during `ModelZoo.load_model()`
- `ModelZoo.load_models()`: for resized/patched models, call
`Predictor.load_model()` with kwarg instead of resp. model name suffix
- `ModelZoo.load_model()`: expect boolean kwargs `patched/resized`
for `wrap_layout_model_patched/resized` model wrappers, respectively
- depending on model type (i.e. size), configure target
batch sizes
- after receiving a prediction task for some model,
look up target batch size, then try to retrieve arrays
from follow-up tasks for the same model on the task queue;
stop when either no tasks are immediately available or
when the combined batch size (input batch size * number of tasks)
reaches the target
- push back tasks for other models to the queue
- rebatch: read all shared arrays, concatenate them along axis 0,
map respective job ids they came from
- predict on new (possibly larger) batch
- split result along axis 0 into number of jobs
- send each result along with its jobid to task queue
- set up a Queue and QueueListener along with ProcessPoolExecutor,
delegating messages from the queue to all handlers
- in forked subprocesses, instead of just inheriting handlers,
replace them with a single QueueHandler, and make sure
log messages get prefixes by the respective job id (img_filename)
so concurrent messages will still be readable
- in the predictor, make sure to pass on the log level to the
spawned subprocess, too
When 338c4a0e wrapped all prediction models for automatic
image size adaptation in CUDA,
- tiling (`_patched`) was indeed faster
- whole (`_resized`) was actually slower
So this reverts the latter part.
- reintroduce ProcessPoolExecutor
(previously for parallel deskewing within pages)
- wrap Eynollah instance into global, so (with forking)
serialization can be avoided – same pattern as in core ocrd.Processor
- move timing/logging into `run_single()`, respectively
- `cache_images()`: only return an image dict (plus extra keys
for file name stem and dpi) - don't set any attributes
- `imread()`: just take from passed image dict, also add `binary` key
- `resize_and_enhance_image_with_column_classifier()`:
* `imread()` from image dict
* set `img_bin` key for binarization result if `input_binary`
* instead of `image_page_org_size` / `page_coord` attributes,
set `img_page` / `coord_page` in image dict
* instead of retval, set `img_res` in image dict
* also set `scale_x` and `scale_y` in image dict, resp.
* simplify
- `resize_image_with_column_classifier()`:
* `imread()` from image dict
* (as in `resize_and_enhance_with_column_classifier`:)
call `calculate_width_height_by_columns_1_2` if `num_col` is
1 or 2 here
* instead of retval, set `img_res` in image dict
* also set `scale_x` and `scale_y` in image dict, resp.
* simplify
- `calculate_width_height_by_columns*()`: simplify, get confidence of
num_col instead of entire array
- `extract_page()`: read `img_res` from image dict; simplify
- `early_page_for_num_of_column_classification()`:
`imread()` from image dict; simplify
- `textline_contours()`: no need for `num_col_classifier` here
- `run_textline()`: no need for `num_col_classifier` here
- `get_regions_light_v()` → `get_regions()`:
* read `img_res` from image dict
* get shapes via `img` from image dict instead of `image_org` attr
* use `img_page` / `coord_page` from image dict instead of attrs
* avoid unnecessary 3-channel arrays
* simplify
- `get_tables_from_model()`: no need for `num_col_classifier` here
- `run_graphics_and_columns_light()` → `run_graphics_and_columns()`:
* pass through image dict instead of `img_bin` (which really was `img_res`)
* simplify
- `run_graphics_and_columns_without_layout()`:
* pass through image dict instead of `img_bin` (which really was `img_res`)
* simplify
- `run_enhancement()`: pass through image dict
- `get_image_and_sclaes*()`: drop
- `run_boxes_full_layout()`:
* pass `image_page` instead of `img_bin` (which really was `image_page`)
* simplify
- `run()`:
* instantiate plotter outside of loop, and independent of img files
* move writer instantiation and overwrite checks into `run_single()`
* add try/catch for `run_single()` w/ logging
- `reset_file_name_dir`: drop
- `run_single()`:
* add some args/kwargs from `run()`
* call `cache_images()` (reading image dict) here
* instantiate writer here instead of (reused) attr in `run()`
* set `scale_x` / `scale_y` in writer from image dict once known
(i.e. after `run_enhancement()`)
* don't return anything, but write PAGE result here
- `check_any_text_region_in_model_one_is_main_or_header_light()` →
`split_textregion_main_vs_header()`
- plotter:
* pass `name` (file stem) from image dict to all methods
* for `write_images_into_directory()`: also `scale_x` and `scale_y`
from image dict
- writer:
* init with width/height
- ocrd processor:
* adapt (just `run_single()` call)
* drop `max_workers=1` restriction (can now run fully parallel)
- `get_textregion_contours_in_org_image_light()` →
`get_textregion_confidences()`:
* take shape from confmat directly instead of extra array
* simplify
When `num_col_classifier` predicted result gets bypassed
by heuristic result from `find_num_col()` (because prediction
had too little confidence or `calculate_width_height_by_columns()`
would have become too large), do not increment `num_col` further
(already 1 more than colseps).
- rename `terminate` → `stopped`
- call `terminate()` from superclass during shutdown
- del `self.model_zoo` in the parent process after spawn,
and in the child during shutdown
- new class `Predictor(multiprocessing.Process)` as stand-in
for EynollahModelZoo:
* calling `load_models()` starts the subprocess (and has
`.model_zoo.load_models()` run internally)
* calling `get()` yields a stand-in that supports `.predict()`,
which actually communicates with the singleton subprocess
via task and result queues, sharing Numpy arrays via SHM
* calling `predict()` with an empty dict (instead of an image)
merely retrieves the respective model's output shapes (cached)
* shared memory objects for arrays are cleared as soon as possible
* log messages are piped through QueueHandler / QueueListener
* exceptions are passed through the queues, and raised afterwards
- move all TF initialization to the predictor
(to avoid back and forth between CPU and GPU memory when looping
over image patches)
- `patch_encoder`: define `Model` subclasses which take an existing
(layout segmentation) model in the constructor, and define a new
`call()` using the existing model in a GPU-only `tf.function`:
* `wrap_layout_model_resized`: just `tf.image.resize()` from
input image to model size, then predict, then resize back
* `wrap_layout_model_patched`: ditto if smaller than model size;
otherwise use `tf.image.extract_patches` for patching in a
sliding-window approach, then predict patches one by one, then
`tf.scatter_nd` to reconstruct to image size
- when compiling `tf.function` graph, make sure to use input signature
with variable image size, but avoid retracing each new size sample
- in `EynollahModelZoo.load_model` for relevant model types,
also wrap the loaded model
* by `wrap_layout_model_resized` under model name + `_resized`
* by `wrap_layout_model_patched` under model name + `_patched`
- introduce `do_prediction_new_concept_autosize`,
replacing `do_prediction/_new_concept`,
but using passed model's `predict` directly without
resizing or tiling to model size
- instead of `do_prediction/_new_concept(True, ...)`,
now call `do_prediction_new_concept_autosize`,
but with `_patched` appended to model name
- instead of `do_prediction/_new_concept(False, ...)`,
now call `do_prediction_new_concept_autosize`,
but with `_resized` appended to model name
- `do_prediction/_new_concept`: avoid unnecessary `np.repeat`
on results, aggregate intermediate artificial class mask and
confidence data in extra arrays
- callers: avoid unnecessary thresholding the result arrays
- callers: adapt (no need to slice into channels)
- simplify by refactoring thresholding and skeletonization into
function `seg_mask_label`
- `extract_text_regions*`: drop unused second result array
- `textline_contours`: avoid calculating unused unpatched prediction
- instead of just comparing the number of connected components,
calculate the GT/pred label incidence matrix and retrieve the
share of singular values (i.e. nearly diagonal under reordering)
over total counts as similarity score
- also, suppress artificial class in that
(Functions cannot be both generators and procedures,
so make this a pure generator and save the image files
on the caller's side; also avoids passing output
directories)
Moreover, simplify by moving the `os.listdir` into the function
body (saving lots of extra variable bindings).
instead of looping over file pairs indefinitely, yielding
Numpy arrays: re-use `keras.utils.image_dataset_from_directory`
here as well, but with img/label generators zipped together
(thus, everything will already be loaded/prefetched on the GPU)
- use relative imports
- use tf.keras everywhere (and ensure v2)
- `weights_ensembling`:
* use `Patches` and `PatchEncoder` from .models
* drop TF1 stuff
* make function / CLI more flexible (expect list of
checkpoint dirs instead of single top-level directory)
- train for `classification`: delegate to `weights_ensembling.run_ensembling`
major conflicts resolved manually:
- branches for non-`light` segmentation already removed in main
- Keras/TF setup and no TF1 sessions, esp. in new ModelZoo
- changes to binarizer and its CLI (`mode`, `overwrite`, `run_single()`)
- writer: `build...` w/ kwargs instead of positional
- training for segmentation/binarization/enhancement tasks:
* drop unused `generate_data_from_folder()`
* simplify `preprocess_imgs()`: turn `preprocess_img()`, `get_patches()`
and `get_patches_num_scale_new()` into generators, only writing
result files in the caller (top-level loop) instead of passing
output directories and file counter
- training for new OCR task:
* `train`: put keys into additional `config_params` where they belong,
resp. (conditioned under existing keys), and w/ better documentation
* `train`: add new keys as kwargs to `run()` to make usable
* `utils`: instead of custom data loader `data_gen_ocr()`, re-use
existing `preprocess_imgs()` (for cfg capture and top-level loop),
but extended w/ new kwargs and calling new `preprocess_img_ocr()`;
the latter as single-image generator (also much simplified)
* `train`: use tf.data loader pipeline from that generator w/ standard
mechanisms for batching, shuffling, prefetching etc.
* `utils` and `train`: instead of `vectorize_label`, use `Dataset.padded_batch`
* add TensorBoard callback and re-use our checkpoint callback
* also use standard Keras top-level loop for training
still problematic (substantially unresolved):
- `Patches` now only w/ fixed implicit size
(ignoring training config params)
- `PatchEncoder` now only w/ fixed implicit num patches and projection dim
(ignoring training config params)
in `return_boxes_of_images_by_order_of_reading_new`,
when the next multicol separator ends in the same column,
do not recurse into subspan if the next starts earlier
(but continue with top span to the right first)
- unify `generate_data_from_folder_training` w/ `..._evaluation`
- instead of recreating array after every batch, just zero out
- cast image results to uint8 instead of uint16
- cast categorical results to float instead of int
- make more config_params keys dependent on each other
- re-order accordingly
- in main, initialise them (as kwarg), so sacred actually
allows overriding them by named config file
- `index_start`: re-introduce cfg key, pass to Keras `Model.fit`
as `initial_epoch`
- make config keys `index_start` and `dir_of_start_model` dependent
on `continue_training`
- improve description
- `utils.provide_patches`: split up loop into
* `utils.preprocess_img` (single img function)
* `utils.preprocess_imgs` (top-level loop)
- capture exceptions for all cases (not just some)
at top level and with informative logging
- avoid repeating / delegating config keys in several
places: only as kwargs to `preprocess_img()`
- read files into memory only once, then re-use
- improve readability (avoiding long lines, repeated code)
when parsing `PrintSpace` or `Border` from PAGE-XML,
- use `lxml` XPath instead of nested loops
- convert points to polygons directly
(instead of painting on canvas and retrieving contours)
- pass result bbox in slice notation
(instead of xywh)
when matching files in `dir_images` by XML path name stem,
* use `dict` instead of `list` to assign reliably
* filter out `.xml` files (so input directories can be mixed)
* show informative warnings for files which cannot be matched
- do not restrict TF version, but depend on tf-keras and
set `TF_USE_LEGACY_KERAS=1` to avoid Keras 3 behaviour
- relax Numpy version requirement up to v2
- relax Torch version requirement
- drop TF1 session management code
- drop TF1 config in favour of TF2 config code for memory growth
- training.*: also simplify and limit line length
- training.train: always train with TensorBoard callback
after selecting the optimum angle on the original
search range, narrow down around in the vicinity
with half the range (adding computational costs,
but gaining precision)
when passing the text region mask, do not apply erosion only
if there are more than 2 columns, but iff `not erosion_hurts`
(consistent with `find_num_col`'s expectations and making
it as easy to find the column gaps on 1 and 2-column pages
as on multi-column pages)
- `find_number_of_columns_in_document`: retain vertical separators
and pass to `find_num_col` for each vertical split
- `return_boxes_of_images_by_order_of_reading_new`: reconstruct
the vertical separators from the segmentation mask and the separator
bboxes; pass it on to `find_num_col` everywhere
- `return_boxes_of_images_by_order_of_reading_new`: no need to
try-catch `find_num_col` anymore
- `return_boxes_of_images_by_order_of_reading_new`: when a vertical
split has too few columns,
* do not raise but lower the threshold `multiplier` responsible for
allowing gaps as column boundaries
* do not pass the `num_col_classifier` (i.e. expected number of
resulting columns) of the entire page to the iterative
`find_num_col` for each existing column, but only the portion
of that span
when searching for gaps between text regions, consider the vertical
separator mask (if given): add the vertical sum of vertical separators
to the peak scores (making column detection more robust if still slighly
skewed or partially obscured by multi-column regions, but fg seps are
present)
- when analysing regions spanning across columns,
disregard tiny regions (smaller than half the median size)
- if a region spans across columns just by a tiny fraction,
and therefore is not good enough for a multi-col separator,
then it should also not be good enough for a multi-col box
maker
- avoid unnecessary `fillPoly` (we already have the mask)
- do not merge hseps if vseps interfere
- remove old criterion (based on total length of hseps)
- create new criterion (no x overlap and x close to each other)
- rename identifiers:
* `sum_dis` → `sum_xspan`
* `diff_max_min_uniques` → `tot_xspan`
* np.std / np.mean → `dev_xspan`
- remove rule cutting around the center of crossing seps
(which is unnecessary and creates small isolated seps
at the center, unrelated to the actual crossing points)
- create rule cutting hseps by vseps _prior_ to merging
- `do_order_of_regions`: simplify aggregating per-box orders
for paragraphs and headings to overall order passed to
`xml_reading_order`; no need for `order_and_id_of_texts`,
no need to return `id_of_texts_tot`
- `do_order_of_regions_with_model`: no need to return `region_ids`
- writer: no need to pass `id_of_texts_tot` in `build_pagexml`
(because the latter does not preserve coordinates;
it scales, even when resizing the image;
this caused coordinate problems when matching deskewed contours)
- reduce `sigma` for smoothing of input to `find_peaks`
(so we get deeper gaps between columns)
- allow column boundaries closer to the margins
(50 instead of 100 or 200 px, 170 instead of 370 px)
- allow column boundaries closer to each other
(300 instead of 400 px)
- add a secondary `grenze` criterion for depth of gap
(relative to lowest minimum, if that is smaller than
the old criterion relative to lowest maximum)
- for calls to `find_num_col` within parts of a page,
do allow unbalanced column boundaries
- rename `return_x_start_end_mothers_childs_and_type_of_reading_order`
→ `return_multicol_separators_x_start_end`, and drop all the analysis
pertaining to mother/child relationships and full-span separators,
also drop the separator unification rules;
instead of the latter, try to combine neighbouring separators more
generally: join column spans iff there is nothing in between
(which also necessitates passing the region mask), and keep only
one of every such redundant pair;
add the top (of each page part) as full-span separator up front,
and return separators already ordered by y
- `return_boxes_of_images_by_order_of_reading_new`:
- also pass regions with separators, so they do not have to be
reconstructed from the separator coordinates, and also contain
images and other non-text region types, when trying to elongate
separators to maximize their span (without introducing overlaps)
- determine connected components of the region mask, i.e. labels
and their respective bboxes, in order to
1. gain additional multi-column separators, if possible
2. avoid cutting through regions which do cross column boundaries
later on
- whenever adding a new bbox, first look up the label map to see if
there are any multi-column regions extending to the right of the
current column; if there are, then advance not just one column
to the right, but as many as necessary to avoid cutting through
these regions
- new core algorithm: iterate separators sorted by y and then column
by column, but whenever the next separator ends in the same column
as the current one or even further left, recurse (i.e. finish that
span first before continuing with the top iteration)
- `lines` → `seps` (to distinguish from textlines)
- `text_regions_p_1_n` → `text_regions_p_d` (because all other
deskewed variables are called like this)
- `pixel` → `label`
- drop connected components analysis to test overlaps between
horizontal separators and (horizontal) neighbours (introduced
in ab17a927)
- instead of converting headings to topline and baseline during
`find_number_of_columns_in_document` (introduced in 9f1595d7),
add them to the matrix unchanged, but mark as extra type
(besides horizontal and vertical separtors)
- convert headings to toplines and baselines no earlier than in
`return_boxes_of_images_by_order_of_reading_new`
- for both headings and horizontal separators, if they already
span multiple columns, check if they would overlap (horizontal)
neighbours by looking at successively larger (left and right)
intervals of columns (and pick the largest elongation which
does not introduce any overlaps)
when y slice (`top:bot`) is not a significant part of the page,
viz. less than 22% (as in `find_number_of_columns_in_document`),
avoid forcing `find_num_col` to reach `num_col_classifier`
(allows large headers not to be split up and thus better ordered)
simplify and document
- simplify
- rename identifiers to make readable:
- `y_sep` → `y_mid` (because the cy gets passed)
- `y_diff` → `y_max` (because the ymax gets passed)
- array instead of list operations
- add docstring and in-line comments
- return (zero-length) numpy array instead of empty list
- when handling lines without mother,
and biggest line already accounts for all columns,
but some are too close to the top and therefore must be removed,
avoid invalidating `biggest` index, causing `IndexError`
- remove try-catch (now unnecessary)
- array instead of list operations
regarding `splitter_y` result, for headings, instead of cutting right
through them via center line, add their toplines and baselines as if
they were horizontal separators
extend horizontal separators to full img width if they do not overlap
any other regions
(only as regards to returned `splitter_y` result,
but without changing returned separators mask)
- `get_textregion_contours_in_org_image_light`: no more need
to also return unchanged contours here (see 41cc38c5); therefore
- `txt_con_org`: no more need for this
(now mere alias to `contours_only_text_parent`); also
- `index_by_text_par_con`: no more need for this (see prev. commit),
so do not pass/return
- `get_slopes_and_deskew_*`: do not pass `contours_only_text`
(where not used)
- `get_slopes_and_deskew_*`: do not return unchanged contours, boxes
- `do_work_of_slopes_*`: adapt respectively
- refactor final `self.full_layout` conditional, removing copied code
- allow running `self.ocr` and `self.tr` branch in both cases (non/fl)
- when running TrOCR, use model / processor / device initialised during init
(instead of ad-hoc loading)
- avoid duplicate and missing mappings by using a different approach:
instead of just minimising the center distance for the N contours
that we expect,
1. get all N:M distances
2. iterate over them from small to large
3. continue adding correspondences until both every original contour
and every deskewed contour have at least one match
4. where one original matches multiple deskewed contours,
join the latter polygons to map as single contour
5. where one deskewed contour matches multiple originals,
split the former by intersecting with each of the latter
(after bringing them into the same coordinate space),
so ultimately only the respective match gets assigned
- when matching undeskewed and new contours, do not just
pick the closest centers, respectively, but also of similar
size (by making the contour area the 3rd dimension of the
vector norm in the distance calculation)
- when searching for boxes matching contour, be more precise:
- avoid heuristic rules ("xmin + 80 within xrange") in favour
of exact criteria (contour properly contained in box)
- for fallback criterion (nearest centers), also require
proper containment of center in box
- `order_of_regions`: remove (now) unnecessary (and insufficient)
workaround for missing indexes (if boxes are not covering contours
exactly)
- array-convert only once (before returning from `order_of_regions`)
- avoid passing `matrix_of_orders` unnecessarily between
`order_of_regions` and `order_and_id_of_texts`
- move `--device` option to group level, apply to all model types (including Torch/ONNX)
- load models w/ `memory_limit` instead of `memory_growth` strategy (faster and less VRAM)
- show full stacktrace in case predictor fails (not just exception name)
- :fire: default to ONNX inference w/ TensorRT instead of TF (much **faster**, but requires **warmup** phase w/ persistent cache directory `$XDG_CONFIG_HOME`)
- :fire: published **new** set of **models**, both for training (TF/Keras) and inference (ONNX) – rebuilt from code changes (see below), *not* retrained
- :fire: Docker image now based on `ocrd/core-cuda-onnx` for layout only, no `[OCR]` in Docker ATM
- OCR: run pages in **parallel** (as for layout) via forking, add `--halt-fail` and `--num-jobs`, too
- improve layout:
* for heuristic reading order, do not try to elongate horizontal separators
* when column classifier is confident enough, do maximally enlarge the image (for 6 columns or more)
- improve TrOCR:
* refactor, simplify
* batch over entire page (faster)
* extract confidence, too
* use beam search instead of greedy decoder
* load model and preprocessor/tokenizer into one object (no need for distinct models)
* no need to resize images in advance
* if set, apply `-nmtc` here, too
* skip lines lower than `-min_conf` instead of setting empty string
- improve Keras OCR:
* refactor, simplify
* batch over entire page (faster)
* use correct confidence estimation
* run binarization ad-hoc (if not provided)
* adapt to all-in-one inference model
* get image size from model (instead of fixed)
* apply `-min_conf` here, too
* skip lines lower than `-min_conf` instead of setting empty string
* separate off `.png` files if `dir_in_bin==dir_in`
* batch flipped line candidates together with normal lines
- training/setup:
* refactor imports from `.models` (single auto-configured `get_model()` call, no `custom_objects` loading)
* drop new setting `reload_weights` in favour of `--rebuild` option for new CLI `eynollah-training convert`
* new CLI for model conversion between Keras (formats HDF5 / native Keras, TF SavedModel), TF-Serving (i.e. `model.export()`) and ONNX
* extract `MusicRegion` from PAGE GT, too
- training/models:
* ViT models: use Keras `Reshape` layer instead of ad-hoc `tf.reshape`
* ViT models: use `tf.map_fn` to iterate over batch in `tf.image.extract_patches` for attention (faster, less VRAM, makes ONNX conversion work)
* Keras (CNN-RNN) OCR backend: replace `Conv1D(channels_first)` (not fully supported by TF/CUDNN on CPU) by `Conv1D(channels_last)` w/ `Permute` layers
* Keras (CNN-RNN) OCR training→inference conversion: encapsulate CTC decoder and inverse string lookup by model itself (no need for extra models and files, all on GPU), always ensemble RGB and binarized input
Added:
- inference backends for TF-Serving and ONNX/TensorRT, differentiate by loaded model type
- integrate training for TrOCR (still untested!)
- integrate weight ensembling for TrOCR
- integrate standalone inference for TrOCR
- OCR-D processor: pass on more parameters:
- `device` selection
- `model_overrides`
- `skip_layout_and_reading_order`
- `num_col_upper`
- `num_col_lower`
- `binarize` (for `input_binary`)
## [0.8.0] - 2026-05-11
* Optimize model performance
* `multiprocessing.SpawnProcess` predictor wrapper for models to have commmunication with Tensorflow in a separate subprocess in a task queue with parallel jobs configurable via `--num-jobs` and maximum number of failed jobs via `--halt-fail`
* Keep batch size low enough for processing fitting into common 8GB GPU (with model-dependent batch resizing prepared but not yet active)
* GPU device can be selected manually with `--device`
* Handle image resizing and tiling in GPU as much as possible to avoid overhead of switching between GPU and CPU
* jit-compile and precompile models where possible (non-autosized, non-patched Keras models)
* Fix bugs and homogenize internal labels related to differing labels for early layout and different stages of full layout detection
* Replace `Lambda` layers with `ZeroPadding2D`, improving size and optimizability of models for `eynollah layout`
* Improved training
* Use connected components for loss function
* Integrate with Tensorboard to observe model training progress, including plots and visualizing intermediate evaluation results
* Simplified model usage
* Models can be overridden individually, so any model trained with `eynollah-training` can replace any model in the [distributions on zenodo](https://zenodo.org/records/17727267)
* `--model` is a CLI option of the `eynollah` root CLI now and should point to the same directory for all subcommands
* Improved reading order detection heuristics
* Improved drop capital, marginalia and column detection
* Fixing bugs in polygon handling and image operations
* No more self-intersecting polygons
* Correct rotation implementation, enlarging/shrinking canvas as necessary
* Use actual area of a polygon instead of length of polygon path or first candidate for comparisons
* Improved PAGE-XML serialization
* Annotate column classifier result in `/PcGts/Page/@custom` (Transkribus convention) and `/PcGts/Metadata/Comment` (QURATOR convention)
* Annotate page skew in `/PcGts/Page/@orientation`
* Calculate and annotate confidences as `Coords/@conf` for regions, lines, images and tables
* Massive refactoring and code quality improvement
* deduplication, idiomatic python, clean parallel processing, class reuse, consistent and meaningful naming
**NOTE** We are aware of a possible issue with regards to the cropping of images. It appears that we have not consistenly cropped images for training. This can lead to suboptimal results for cropped images. If you experience quality issues with the `eynollah layout`, try setting the `-ipe/--ignore_page_extraction` option to skip the builtin cropping. We will rectify this in the next trainings.
## [0.7.0] - 2026-01-30
Added:
* "Model zoo", central place to describe and load models, #207
* Training code for the CNN/RNN OCR model
Changed:
* Lint training code, #204
* Update documentation: README, pyproject.toml metadata, guides in `docs/`, #209
## [0.6.0] - 2025-10-17
Added:
* `eynollah-training` CLI and docs for training the models, #187, #193, https://github.com/qurator-spk/sbb_pixelwise_segmentation/tree/unifying-training-models
Fixed:
* `join_polygons` always returning Polygon, not MultiPolygon, #203
## [0.6.0rc2] - 2025-10-14
Fixed:
* Prevent OOM GPU error by avoiding loading the `region_fl` model, #199
* XML output: encoding should be `utf-8`, not `utf8`, #196, #197
## [0.6.0rc1] - 2025-10-10
Fixed:
* continue processing when no columns detected but text regions exist
* convert marginalia to main text if no main text is present
* reset deskewing angle to 0° when text covers <30%imageareaanddetectedangle>45°
* Drop capitals are now handled separately from their corresponding textline
* Marginals are now divided into left and right. Their reading order is written first for left marginals, then for right marginals, and within each side from top to bottom
* Added a new page extraction model. Instead of bounding boxes, it outputs page contours in the XML file, improving results for skewed pages
* Improved reading order for cases where a textline is segmented into multiple smaller textlines
Changed
* CLIs: read only allowed filename suffixes (image or XML) with `--dir_in`
* CLIs: make all output option required, and `-i` / `-di` required but mutually exclusive
* ocr CLI: drop redundant `-brb` in favour of just `-dib`
* APIs: move all input/output path options from class (kwarg and attribute) ro `run` kwarg
* layout textlines: polygonal also without `-cl`
Added:
* `eynollah machine-based-reading-order` CLI to run reading order detection, #175
* `eynollah enhancement` CLI to run image enhancement, #175
* Improved models for page extraction and reading order detection, #175
* For the lightweight version (layout and textline detection), thresholds are now assigned to the artificial class. Users can apply these thresholds to improve detection of isolated textlines and regions. To counteract the drawback of thresholding, the skeleton of the artificial class is used to keep lines as thin as possible (resolved issues #163 and #161)
* Added and integrated a trained CNN-RNN OCR models
* Added and integrated a trained TrOCR model
* Improved OCR detection to support vertical and curved textlines
* Introduced a new machine-based reading order model with rotation augmentation
* Optimized reading order speed by clustering text regions that belong to the same block, maintaining top-to-bottom order
* Implemented text merging across textlines based on hyphenation when a line ends with a hyphen
* Integrated image enhancement as a separate use case
* Added reading order functionality on the layout level as a separate use case
* CNN-RNN OCR models provide confidence scores for predictions
* Added OCR visualization: predicted OCR can be overlaid on an image of the same size as the input
* Introduced a threshold value for CNN-RNN OCR models, allowing users to filter out low-confidence textline predictions
* For OCR, users can specify a single model by name instead of always using the default model
* Under the OCR use case, if Ground Truth XMLs and images are available, textline image and corresponding text extraction can now be performed
Merged PRs:
* better machine based reading order + layout and textline + ocr by @vahidrezanezhad in https://github.com/qurator-spk/eynollah/pull/175
* CI: pypi by @kba in https://github.com/qurator-spk/eynollah/pull/154
* CI: Use most recent actions/setup-python@v5 by @kba in https://github.com/qurator-spk/eynollah/pull/157
* update docker by @bertsky in https://github.com/qurator-spk/eynollah/pull/159
* Ocrd fixes by @kba in https://github.com/qurator-spk/eynollah/pull/167
* Updating readme for eynollah use cases cli by @kba in https://github.com/qurator-spk/eynollah/pull/166
* OCR-D processor: expose reading_order_machine_based by @bertsky in https://github.com/qurator-spk/eynollah/pull/171
* prepare release v0.5.0: fix logging by @bertsky in https://github.com/qurator-spk/eynollah/pull/180
* mb_ro_on_layout: remove copy-pasta code not actually used by @kba in https://github.com/qurator-spk/eynollah/pull/181
* prepare release v0.5.0: improve CLI docstring, refactor I/O path options from class to run kwargs, increase test coverage @bertsky in #182
* prepare release v0.5.0: fix for OCR doit subtest by @bertsky in https://github.com/qurator-spk/eynollah/pull/183
* Prepare release v0.5.0 by @kba in https://github.com/qurator-spk/eynollah/pull/178
* updating eynollah README, how to use it for use cases by @vahidrezanezhad in https://github.com/qurator-spk/eynollah/pull/156
* add feedback to command line interface by @michalbubula in https://github.com/qurator-spk/eynollah/pull/170
:warning: Development is currently focused on achieving the best possible quality of results for a wide variety of historical documents and therefore processing can be very slow. We aim to improve this, but contributions are welcome.
:warning: Development is focused on achieving the best quality of results for a wide variety of historical
documents using a combination of multiple deep learning models and heuristics; therefore processing can be slow.
## Installation
Python `3.8-3.11` with Tensorflow `<2.13` on Linux are currently supported.
For (limited) GPU support the CUDA toolkit needs to be installed.
Python `3.8-3.11` with ONNX Runtime on Linux are currently supported.
For GPU support, NVidia drivers supporting CUDA 12 must be installed.
The runtime dependencies will pull in ONNX, TensorRT and CUDA runtime
libraries (including cuDNN) from PyPI.
You can either install from PyPI
@ -41,78 +47,192 @@ cd eynollah; pip install -e .
Alternatively, you can run `make install` or `make install-dev` for editable installation.
To also install the dependencies for the OCR engines:
```
pip install "eynollah[OCR]"
# or
make install EXTRAS=OCR
```
> **Note**: Requirements for OCR are more involved,
> as they may need Tensorflow (with tf-keras) and/or
> Torch (with transformers). Those two frameworks may
> also have conflicting CUDA dependencies. An ONNX
> conversion for these models may be achieved soon.
> :construction:
### Docker
Use
```
docker pull ghcr.io/qurator-spk/eynollah:latest
```
When using Eynollah with Docker, see [`docker.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/docker.md).
## Models
Pre-trained models can be downloaded from [qurator-data.de](https://qurator-data.de/eynollah/) or [huggingface](https://huggingface.co/SBB?search_models=eynollah).
For documentation on methods and models, have a look at [`models.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/models.md).
Pretrained models can be downloaded from [Zenodo](https://doi.org/10.5281/zenodo.17194823) or [Hugging Face](https://huggingface.co/SBB?search_models=eynollah).
## Train
In case you want to train your own model with Eynollah, have a look at [`train.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/train.md).
For fast runtime inference, download the ONNX models distributed as `models_inference_...zip`.
For finetuning training, download the original (Tensorflow / Torch) models distributed as `models_training...zip`
(and install the `[training]` extra).
For model documentation and model cards, see [`models.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/models.md).
## Training
To train your own model with Eynollah, see [`train.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/train.md) and use the tools in the [`train`](https://github.com/qurator-spk/eynollah/tree/main/train) folder.
## Usage
The command-line interface can be called like this:
Model card: [Image Enhancement](https://huggingface.co/SBB/eynollah-enhancement)
This model addresses image resolution, specifically targeting documents with suboptimal resolution. In instances where
@ -30,12 +33,14 @@ the detection of document layout exhibits inadequate performance, the proposed e
the quality and clarity of the images, thus facilitating enhanced visual interpretation and analysis.
### Page extraction / border detection
Model card: [Page Extraction/Border Detection](https://huggingface.co/SBB/eynollah-page-extraction)
A problem that can negatively affect OCR are black margins around a page caused by document scanning. A deep learning
model helps to crop to the page borders by using a pixel-wise segmentation method.
### Column classification
Model card: [Column Classification](https://huggingface.co/SBB/eynollah-column-classifier)
This model is a trained classifier that recognizes the number of columns in a document by use of a training set with
@ -43,6 +48,7 @@ manual classification of all documents into six classes with either one, two, th
respectively.
### Binarization
Model card: [Binarization](https://huggingface.co/SBB/eynollah-binarization)
This model is designed to tackle the intricate task of document image binarization, which involves segmentation of the
@ -52,6 +58,7 @@ capability of the model enables improved accuracy and reliability in subsequent
enhanced document understanding and interpretation.
### Main region detection
Model card: [Main Region Detection](https://huggingface.co/SBB/eynollah-main-regions)
This model has employed a different set of labels, including an artificial class specifically designed to encompass the
@ -61,6 +68,7 @@ during the inference phase. By incorporating this methodology, improved efficien
model's ability to accurately identify and classify text regions within documents.
### Main region detection (with scaling augmentation)
Model card: [Main Region Detection (with scaling augmentation)](https://huggingface.co/SBB/eynollah-main-regions-aug-scaling)
Utilizing scaling augmentation, this model leverages the capability to effectively segment elements of extremely high or
@ -69,12 +77,14 @@ categorizing and isolating such elements, thereby enhancing its overall performa
documents with varying scale characteristics.
### Main region detection (with rotation augmentation)
Model card: [Main Region Detection (with rotation augmentation)](https://huggingface.co/SBB/eynollah-main-regions-aug-rotation)
This model takes advantage of rotation augmentation. This helps the tool to segment the vertical text regions in a
robust way.
### Main region detection (ensembled)
Model card: [Main Region Detection (ensembled)](https://huggingface.co/SBB/eynollah-main-regions-ensembled)
The robustness of this model is attained through an ensembling technique that combines the weights from various epochs.
@ -82,16 +92,19 @@ By employing this approach, the model achieves a high level of resilience and st
strengths of multiple epochs to enhance its overall performance and deliver consistent and reliable results.
### Full region detection (1,2-column documents)
Model card: [Full Region Detection (1,2-column documents)](https://huggingface.co/SBB/eynollah-full-regions-1column)
This model deals with documents comprising of one and two columns.
### Full region detection (3,n-column documents)
Model card: [Full Region Detection (3,n-column documents)](https://huggingface.co/SBB/eynollah-full-regions-3pluscolumn)
This model is responsible for detecting headers and drop capitals in documents with three or more columns.
### Textline detection
Model card: [Textline Detection](https://huggingface.co/SBB/eynollah-textline)
The method for textline detection combines deep learning and heuristics. In the deep learning part, an image-to-image
@ -106,6 +119,7 @@ segmentation is first deskewed and then the textlines are separated with the sam
textline bounding boxes. Later, the strap is rotated back into its original orientation.
### Textline detection (light)
Model card: [Textline Detection Light (simpler but faster method)](https://huggingface.co/SBB/eynollah-textline_light)
The method for textline detection combines deep learning and heuristics. In the deep learning part, an image-to-image
@ -119,6 +133,7 @@ enhancing the model's ability to accurately identify and delineate individual te
eliminates the need for additional heuristics in extracting textline contours.
### Table detection
Model card: [Table Detection](https://huggingface.co/SBB/eynollah-tables)
The objective of this model is to perform table segmentation in historical document images. Due to the pixel-wise
@ -128,20 +143,84 @@ effectively identify and delineate tables within the historical document images,
enabling subsequent analysis and interpretation.
### Image detection
Model card: [Image Detection](https://huggingface.co/SBB/eynollah-image-extraction)
This model is used for the task of illustration detection only.
### Reading order detection
Model card: [Reading Order Detection]()
TODO
The model extracts the reading order of text regions from the layout by classifying pairwise relationships between them. A sorting algorithm then determines the overall reading sequence.
### OCR
We have trained three OCR models: two CNN-RNN–based models and one transformer-based TrOCR model. The CNN-RNN models are generally faster and provide better results in most cases, though their performance decreases with heavily degraded images. The TrOCR model, on the other hand, is computationally expensive and slower during inference, but it can possibly produce better results on strongly degraded images.
Compared to the model_eynollah_ocr_cnnrnn_20250805 model, this model is trained on a larger proportion of Antiqua data and achieves superior performance.
Additionally, some heuristic methods are employed to further improve the model predictions:
* After border detection, the largest contour is determined by a bounding box, and the image cropped to these coordinates.
* For text region detection, the image is scaled up to make it easier for the model to detect background space between text regions.
* Unlike the non-light version, where the image is scaled up to help the model better detect the background spaces between text regions, the light version uses down-scaled images. In this case, introducing an artificial class along the boundaries of text regions and text lines has helped to isolate and separate the text regions more effectively.
* A minimum area is defined for text regions in relation to the overall image dimensions, so that very small regions that are noise can be filtered out.
* Deskewing is applied on the text region level (due to regions having different degrees of skew) in order to improve the textline segmentation result.
* After deskewing, a calculation of the pixel distribution on the X-axis allows the separation of textlines (foreground) and background pixels.
* Finally, using the derived coordinates, bounding boxes are determined for each textline.
* In the non-light version, deskewing is applied at the text-region level (since regions may have different degrees of skew) to improve text-line segmentation results. In contrast, the light version performs deskewing only at the page level to enhance margin detection and heuristic reading-order estimation.
* After deskewing, a calculation of the pixel distribution on the X-axis allows the separation of textlines (foreground) and background pixels (only in non-light version).
* Finally, using the derived coordinates, bounding boxes are determined for each textline (only in non-light version).
* As mentioned above, the reading order can be determined using a model; however, this approach is computationally expensive, time-consuming, and less accurate due to the limited amount of ground-truth data available for training. Therefore, our tool uses a heuristic reading-order detection method as the default. The heuristic approach relies on headers and separators to determine the reading order of text regions.
A small sample of training data for binarization experiment can be found on [Zenodo](https://zenodo.org/records/17243320/files/training_data_sample_binarization_v0_5_1.tar.gz?download=1),
> Tool to extract 2-D or 3-D RGB images from PAGE-XML data. In the former case, the output will be 1 2-D image array which each class has filled with a pixel value. In the case of a 3-D RGB image,
each class will be defined with a RGB value and beside images, a text file of classes will also be produced.
> Extract region classes and their colours in mask (pseg) images. Allows the color map as free dict parameter, and comes with a default that mimics PageViewer's coloring for quick debugging; it also warns when regions do overlap.
# Training documentation
This aims to assist users in preparing training datasets, training models, and performing inference with trained models.
We cover various use cases including pixel-wise segmentation, image classification, image enhancement, and machine-based
reading order detection. For each use case, we provide guidance on how to generate the corresponding training dataset.
The following three tasks can all be accomplished using the code in the
and 'catch-word'. The known types for graphic regions are 'handwritten-annotation', 'decoration', 'stamp', and
'signature'.
Since we don't know all types of text and graphic regions, unknown cases can arise. To handle these, we have defined
two additional types, "rest_as_paragraph" and "rest_as_decoration", to ensure that no unknown types are missed.
This way, users can extract all known types from the labels and be confident that no unknown types are overlooked.
Moreover, text regions and graphic regions in particular are subdivided via `@type`:
- The allowed subtypes for text regions are `paragraph`, `heading`, `marginalia`, `drop-capital`, `header`, `footnote`,
`footnote-continued`, `signature-mark`, `page-number` and `catch-word`.
- The known subtypes for graphic regions are `handwritten-annotation`, `decoration`, `stamp` and `signature`.
In the custom JSON file shown above, "header" and "heading" are extracted as the same class, while "marginalia" is shown
as a different class. All other text region types, including "drop-capital," are grouped into the same class. For the
graphic region, "stamp" has its own class, while all other types are classified together. "Image region" and "separator
region" are also present in the label. However, other regions like "noise region" and "table region" will not be
included in the label PNG file, even if they have information in the page XML files, as we chose not to include them.
These types and subtypes must be mapped to classes for the segmentation model. However, sometimes these fine-grained
distinctions are not useful or the existing annotations are not very usable (too scarce or too unreliable).
In that case, instead of these subtypes with a specific mapping, they can be pooled together by using the two special
types:
- `rest_as_paragraph` (mapping missing TextRegion subtypes and `paragraph`)
- `rest_as_decoration` (mapping missing GraphicRegion subtypes and `decoration`)
`python generate_gt_for_training.py pagexml2label -dx "dir of GT xml files" -do "dir where output label png files will
be written" -cfg "custom config json file" -to "output type which has 2d and 3d. 2d is used for training and 3d is just
to visualise the labels" "`
(That way, users can extract all known types from the labels and be confident that no subtypes are overlooked.)
We have also defined an artificial class that can be added to the boundary of text region types or text lines. This key
is called "artificial_class_on_boundary." If users want to apply this to certain text regions in the layout use case,
the example JSON config file should look like this:
In the custom JSON example shown above, `header` and `heading` are extracted as the same class,
while `marginalia` is modelled as a different class. All other text region types, including `drop-capital`,
are grouped into the same class. For graphic regions, `stamp` has its own class, while all other types
are classified together. `ImageRegion` and `SeparatorRegion` will also represented with a class label in the
training data. However, other regions like `NoiseRegion` or `TableRegion` will not be included in the PNG files,
even if they were present in the PAGE XML.
The tool expects various command-line options:
```sh
eynollah-training generate-gt pagexml2label \
-dx "dir of input PAGE XML files" \
-do "dir of output label PNG files" \
-cfg "custom config JSON file" \
-to "output type (2d or 3d)"
```
As output type, use
- `2d` for training,
- `3d` to just visualise the labels.
We have also defined an artificial class that can be added to (rendered around) the boundary
of text region types or text lines in order to make separation of neighbouring segments more
reliable. The key is called `artificial_class_on_boundary`, and it takes a list of text region
types to be applied to.
Our example JSON config file could then look like this:
```yaml
{
@ -147,14 +237,15 @@ the example JSON config file should look like this:
}
```
This implies that the artificial class label, denoted by 7, will be present on PNG files and will only be added to the
elements labeled as "paragraph," "header," "heading," and "marginalia."
This implies that the artificial class label (denoted by 7) will be present in the generated PNG files
and will only be added around segments labeled `paragraph`, `header`, `heading` or `marginalia`. (This
class will be handled specially during decoding at inference, and not show up in final results.)
For "textline", "word", and "glyph", the artificial class on the boundaries will be activated only if the
"artificial_class_label" key is specified in the config file. Its value should be set as 2 since these elements
represent binary cases. For example, if the background and textline are denoted as 0 and 1 respectively, then the
artificial class should be assigned the value 2. The example JSON config file should look like this for "textline" use
case:
For `printspace`, `textline`, `word`, and `glyph` segmentation use-cases, there is no `artificial_class_on_boundary` key,
but `artificial_class_label` is available. If specified in the config file, then its value should be set at 2, because
these elements represent binary classification problems (with background represented as 0, and segments as 1, respectively).
For example, the JSON config for textline detection could look as follows:
```yaml
{
@ -163,26 +254,38 @@ case:
}
```
If the coordinates of "PrintSpace" or "Border" are present in the page XML ground truth files, and the user wishes to
crop only the print space area, this can be achieved by activating the "-ps" argument. However, it should be noted that
in this scenario, since cropping will be applied to the label files, the directory of the original images must be
provided to ensure that they are cropped in sync with the labels. This ensures that the correct images and labels
required for training are obtained. The command should resemble the following:
If the coordinates of `PrintSpace` (or `Border`) are present in the PAGE XML ground truth files,
and one wishes to crop images to only cover the print space bounding box, this can be achieved
by passing the `-ps` option. Note that in this scenario, the directory of the original images
must also be provided, to ensure that the images are cropped in sync with the labels. The command
line would then resemble this:
`python generate_gt_for_training.py pagexml2label -dx "dir of GT xml files" -do "dir where output label png files will
be written" -cfg "custom config json file" -to "output type which has 2d and 3d. 2d is used for training and 3d is just
to visualise the labels" -ps -di "dir where the org images are located" -doi "dir where the cropped output images will
be written" `
```sh
eynollah-training generate-gt pagexml2label \
-dx "dir of input PAGE XML files" \
-do "dir of output label PNG files" \
-cfg "custom config JSON file" \
-to "output type (2d or 3d)" \
-ps \
-di "dir of input original images" \
-doi "dir of output cropped images"
```
Also, note that it can be detrimental to layout training if there are visible segments which
the annotation does not account for (and thus the model must learn to ignore). So if the images
are not cropped, the `-ps`_should_ be used. If a PAGE XML file is missing `PrintSpace` (or `Border`)
annotations, use `-mps` to either `skip` these or `project` (i.e. crop from existing segments).
## Train a model
### classification
For the classification use case, we haven't provided a ground truth generator, as it's unnecessary. For classification,
all we require is a training directory with subdirectories, each containing images of its respective classes. We need
separate directories for training and evaluation, and the class names (subdirectories) must be consistent across both
directories. Additionally, the class names should be specified in the config JSON file, as shown in the following
example. If, for instance, we aim to classify "apple" and "orange," with a total of 2 classes, the
"classification_classes_name" key in the config file should appear as follows:
For the image classification use-case, we have not provided a ground truth generator, as it is unnecessary.
All we require is a training directory with subdirectories, each containing images of its respective classes. We need
separate directories for training and evaluation, and the class names (subdirectories) must be consistent across both
directories. Additionally, the class names should be specified in the config JSON file, as shown in the following
example. If, for instance, we aim to classify "apple" and "orange," with a total of 2 classes, the
`classification_classes_name` key in the config file should appear as follows:
```yaml
{
@ -204,18 +307,18 @@ example. If, for instance, we aim to classify "apple" and "orange," with a total
}
```
The "dir_train" should be like this:
Then `dir_train` should be like this:
```
```
.
└── train # train directory
├── apple # directory of images for apple class
└── orange # directory of images for orange class
```
And the "dir_eval" the same structure as train directory:
And `dir_eval` analogously:
```
```
.
└── eval # evaluation directory
├── apple # directory of images for apple class
@ -225,11 +328,13 @@ And the "dir_eval" the same structure as train directory:
The classification model can be trained using the following command line:
`python train.py with config_classification.json`
```sh
eynollah-training train with config_classification.json
```
As evident in the example JSON file above, for classification, we utilize a "f1_threshold_classification" parameter.
This parameter is employed to gather all models with an evaluation f1 score surpassing this threshold. Subsequently,
an ensemble of these model weights is executed, and a model is saved in the output directory as "model_ens_avg".
As evident in the example JSON file above, for classification, we utilize a "f1_threshold_classification" parameter.
This parameter is employed to gather all models with an evaluation f1 score surpassing this threshold. Subsequently,
an ensemble of these model weights is executed, and a model is saved in the output directory as "model_ens_avg".
Additionally, the weight of the best model based on the evaluation f1 score is saved as "model_best".
### reading order
@ -271,67 +376,91 @@ And the "dir_eval" the same structure as train directory:
└── labels # directory of labels
```
The classification model can be trained like the classification case command line.
The reading-order model can be trained like the classification case command line.
### Segmentation (Textline, Binarization, Page extraction and layout) and enhancement
#### Parameter configuration for segmentation or enhancement usecases
The following parameter configuration can be applied to all segmentation use cases and enhancements. The augmentation,
its sub-parameters, and continued training are defined only for segmentation use cases and enhancements, not for
The following parameter configuration can be applied to all segmentation use cases and enhancements. The augmentation,
its sub-parameters, and continued training are defined only for segmentation use cases and enhancements, not for
classification and machine-based reading order, as you can see in their example config files.
* backbone_type: For segmentation tasks (such as text line, binarization, and layout detection) and enhancement, we
* offer two backbone options: a "nontransformer" and a "transformer" backbone. For the "transformer" backbone, we first
* apply a CNN followed by a transformer. In contrast, the "nontransformer" backbone utilizes only a CNN ResNet-50.
* task : The task parameter can have values such as "segmentation", "enhancement", "classification", and "reading_order".
* patches: If you want to break input images into smaller patches (input size of the model) you need to set this
* parameter to ``true``. In the case that the model should see the image once, like page extraction, patches should be
* set to ``false``.
* n_batch: Number of batches at each iteration.
* n_classes: Number of classes. In the case of binary classification this should be 2. In the case of reading_order it
* should set to 1. And for the case of layout detection just the unique number of classes should be given.
* n_epochs: Number of epochs.
* input_height: This indicates the height of model's input.
* input_width: This indicates the width of model's input.
* weight_decay: Weight decay of l2 regularization of model layers.
* pretraining: Set to ``true`` to load pretrained weights of ResNet50 encoder. The downloaded weights should be saved
* in a folder named "pretrained_model" in the same directory of "train.py" script.
* augmentation: If you want to apply any kind of augmentation this parameter should first set to ``true``.
* flip_aug: If ``true``, different types of filp will be applied on image. Type of flips is given with "flip_index" parameter.
* blur_aug: If ``true``, different types of blurring will be applied on image. Type of blurrings is given with "blur_k" parameter.
* scaling: If ``true``, scaling will be applied on image. Scale of scaling is given with "scales" parameter.
* degrading: If ``true``, degrading will be applied to the image. The amount of degrading is defined with "degrade_scales" parameter.
* brightening: If ``true``, brightening will be applied to the image. The amount of brightening is defined with "brightness" parameter.
* rotation_not_90: If ``true``, rotation (not 90 degree) will be applied on image. Rotation angles are given with "thetha" parameter.
* rotation: If ``true``, 90 degree rotation will be applied on image.
* binarization: If ``true``,Otsu thresholding will be applied to augment the input data with binarized images.
* scaling_bluring: If ``true``, combination of scaling and blurring will be applied on image.
* scaling_binarization: If ``true``, combination of scaling and binarization will be applied on image.
* scaling_flip: If ``true``, combination of scaling and flip will be applied on image.
* flip_index: Type of flips.
* blur_k: Type of blurrings.
* scales: Scales of scaling.
* brightness: The amount of brightenings.
* thetha: Rotation angles.
* degrade_scales: The amount of degradings.
* continue_training: If ``true``, it means that you have already trained a model and you would like to continue the training. So it is needed to provide the dir of trained model with "dir_of_start_model" and index for naming the models. For example if you have already trained for 3 epochs then your last index is 2 and if you want to continue from model_1.h5, you can set ``index_start`` to 3 to start naming model with index 3.
* weighted_loss: If ``true``, this means that you want to apply weighted categorical_crossentropy as loss fucntion. Be carefull if you set to ``true``the parameter "is_loss_soft_dice" should be ``false``
* data_is_provided: If you have already provided the input data you can set this to ``true``. Be sure that the train and eval data are in "dir_output". Since when once we provide training data we resize and augment them and then we write them in sub-directories train and eval in "dir_output".
* dir_train: This is the directory of "images" and "labels" (dir_train should include two subdirectories with names of images and labels ) for raw images and labels. Namely they are not prepared (not resized and not augmented) yet for training the model. When we run this tool these raw data will be transformed to suitable size needed for the model and they will be written in "dir_output" in train and eval directories. Each of train and eval include "images" and "labels" sub-directories.
* index_start: Starting index for saved models in the case that "continue_training" is ``true``.
* dir_of_start_model: Directory containing pretrained model to continue training the model in the case that "continue_training" is ``true``.
* transformer_num_patches_xy: Number of patches for vision transformer in x and y direction respectively.
* transformer_patchsize_x: Patch size of vision transformer patches in x direction.
* transformer_patchsize_y: Patch size of vision transformer patches in y direction.
* transformer_projection_dim: Transformer projection dimension. Default value is 64.
* transformer_mlp_head_units: Transformer Multilayer Perceptron (MLP) head units. Default value is [128, 64].
* transformer_layers: transformer layers. Default value is 8.
* transformer_num_heads: Transformer number of heads. Default value is 4.
* transformer_cnn_first: We have two types of vision transformers. In one type, a CNN is applied first, followed by a transformer. In the other type, this order is reversed. If transformer_cnn_first is true, it means the CNN will be applied before the transformer. Default value is true.
* `task`: The task parameter must be one of the following values:
- `binarization`,
- `enhancement`,
- `segmentation`,
- `classification`,
- `reading_order`.
* `backbone_type`: For the tasks `segmentation` (such as text line, and region layout detection),
`binarization` and `enhancement`, we offer two backbone options:
- `nontransformer` (only a CNN ResNet-50).
- `transformer` (first apply a CNN, followed by a transformer)
* `transformer_cnn_first`: Whether to apply the CNN first (followed by the transformer) when using `transformer` backbone.
* `transformer_num_patches_xy`: Number of patches for vision transformer in x and y direction respectively.
* `transformer_patchsize_x`: Patch size of vision transformer patches in x direction.
* `transformer_patchsize_y`: Patch size of vision transformer patches in y direction.
* `transformer_projection_dim`: Transformer projection dimension. Default value is 64.
* `transformer_mlp_head_units`: Transformer Multilayer Perceptron (MLP) head units. Default value is [128, 64].
* `transformer_layers`: transformer layers. Default value is 8.
* `transformer_num_heads`: Transformer number of heads. Default value is 4.
* `patches`: Whether to break up (tile) input images into smaller patches (input size of the model).
If `false`, the model will see the image once (resized to the input size of the model).
Should be set to `false` for cases like page extraction.
* `n_batch`: Number of batches at each iteration.
* `n_classes`: Number of classes. In the case of binary classification this should be 2. In the case of reading_order it
should set to 1. And for the case of layout detection just the unique number of classes should be given.
* `n_epochs`: Number of epochs (iterations over the data) to train.
* `input_height`: the image height for the model's input.
* `input_width`: the image width for the model's input.
* `weight_decay`: Weight decay of l2 regularization of model layers.
* `weighted_loss`: If `true`, this means that you want to apply weighted categorical crossentropy as loss function.
(Mutually exclusive with `is_loss_soft_dice`, and only applies for `segmentation` and `binarization` tasks.)
* `pretraining`: Set to `true` to (download and) initialise pretrained weights of ResNet50 encoder.
* `dir_train`: Path to directory of raw training data (as extracted via `pagexml2labels`, i.e. with subdirectories
`images` and `labels` for input images and output labels.
(These are not prepared for training the model, yet. Upon first run, the raw data will be transformed to suitable size
needed for the model, and written in `dir_output` under `train` and `eval` subdirectories. See `data_is_provided`.)
* `dir_eval`: Ditto for raw evaluation data.
* `dir_output`: Directory to write model checkpoints, logs (for Tensorboard) and precomputed images to.
* `data_is_provided`: If you have already trained at least one complete epoch (using the same data settings) before,
you can set this to `true` to avoid computing the resized / patched / augmented image files again.
Be sure that there are subdirectories `train` and `eval` data are in `dir_output` (each with subdirectories `images`
and `labels`, respectively).
* `continue_training`: If `true`, continue training a model checkpoint from a previous run.
This requires providing the directory of the model checkpoint to load via `dir_of_start_model`
and setting `index_start` counter for naming new checkpoints.
For example if you have already trained for 3 epochs, then your last index is 2, so if you want
to continue with `model_04`, `model_05` etc., set `index_start=3`.
* `index_start`: Starting index for saving models in the case that `continue_training` is `true`.
(Existing checkpoints above this will be overwritten.)
* `dir_of_start_model`: Directory containing existing model checkpoint to initialise model weights from when `continue_training=true`.
(Can be an epoch-interval checkpoint, or batch-interval checkpoint from `save_interval`.)
* `augmentation`: If you want to apply any kind of augmentation this parameter should first set to `true`.
The remaining settings pertain to that...
* `flip_aug`: If `true`, different types of flipping over the image arrays. Requires `flip_index` parameter.
* `flip_index`: List of flip codes (as in `cv2.flip`, i.e. 0 for vertical, positive for horizontal shift, negative for vertical and horizontal shift).
* `blur_aug`: If `true`, different types of blurring will be applied on image. Requires `blur_k` parameter.
* `blur_k`: Method of blurring (`gauss`, `median` or `blur`).
* `scaling`: If `true`, scaling will be applied on image. Requires `scales` parameter.
* `scales`: List of scale factors for scaling.
* `scaling_bluring`: If `true`, combination of scaling and blurring will be applied on image.
* `scaling_binarization`: If `true`, combination of scaling and binarization will be applied on image.
* `scaling_flip`: If `true`, combination of scaling and flip will be applied on image.
* `degrading`: If `true`, degrading will be applied to the image. Requires `degrade_scales` parameter.
* `degrade_scales`: List of intensity factors for degrading.
* `brightening`: If `true`, brightening will be applied to the image. Requires `brightness` parameter.
* `brightness`: List of intensity factors for brightening.
* `binarization`: If `true`, Otsu thresholding will be applied to augment the input data with binarized images.
* `dir_img_bin`: With `binarization`, use this directory to read precomputed binarized images instead of ad-hoc Otsu.
(Base names should correspond to the files in `dir_train/images`.)
* `rotation`: If `true`, 90° rotation will be applied on images.
* `rotation_not_90`: If `true`, random rotation (other than 90°) will be applied on image. Requires `thetha` parameter.
* `thetha`: List of rotation angles (in degrees).
In the case of segmentation and enhancement the train and evaluation directory should be as following.
In case of segmentation and enhancement the train and evaluation data should be organised as follows.
The "dir_train" should be like this:
The "dir_train" directory should be like this:
```
.
@ -349,12 +478,40 @@ And the "dir_eval" the same structure as train directory:
└── labels # directory of labels
```
After configuring the JSON file for segmentation or enhancement, training can be initiated by running the following
command, similar to the process for classification and reading order:
After configuring the JSON file for segmentation or enhancement,
training can be initiated by running the following command line,
similar to classification and reading-order model training:
`python train.py with config_classification.json`
```sh
eynollah-training train with config_classification.json
```
#### Binarization
### Ground truth format
Lables for each pixel are identified by a number. So if you have a
binary case, ``n_classes`` should be set to ``2`` and labels should
be ``0`` and ``1`` for each class and pixel.
In the case of multiclass, just set ``n_classes`` to the number of classes
you have and the try to produce the labels by pixels set from ``0 , 1 ,2 .., n_classes-1``.
The labels format should be png.
Our lables are 3 channel png images but only information of first channel is used.
If you have an image label with height and width of 10, for a binary case the first channel should look like this:
Label: [ [1, 0, 0, 1, 1, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
...,
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ]
This means that you have an image by `10*10*3` and `pixel[0,0]` belongs
to class `1` and `pixel[0,1]` belongs to class `0`.
A small sample of training data for binarization experiment can be found here, [Training data sample](https://qurator-data.de/~vahid.rezanezhad/binarization_training_data_sample/), which contains images and lables folders.
An example config json file for binarization can be like this:
```yaml
@ -398,7 +555,7 @@ An example config json file for binarization can be like this:
"thetha" : [10, -10],
"continue_training": false,
"index_start" : 0,
"dir_of_start_model" : " ",
"dir_of_start_model" : " ",
"weighted_loss": false,
"is_loss_soft_dice": false,
"data_is_provided": false,
@ -443,7 +600,7 @@ An example config json file for binarization can be like this:
"thetha" : [10, -10],
"continue_training": false,
"index_start" : 0,
"dir_of_start_model" : " ",
"dir_of_start_model" : " ",
"weighted_loss": false,
"is_loss_soft_dice": false,
"data_is_provided": false,
@ -488,7 +645,7 @@ An example config json file for binarization can be like this:
"thetha" : [10, -10],
"continue_training": false,
"index_start" : 0,
"dir_of_start_model" : " ",
"dir_of_start_model" : " ",
"weighted_loss": false,
"is_loss_soft_dice": false,
"data_is_provided": false,
@ -498,7 +655,7 @@ An example config json file for binarization can be like this:
}
```
It's important to mention that the value of n_classes for enhancement should be 3, as the model's output is a 3-channel
It's important to mention that the value of n_classes for enhancement should be 3, as the model's output is a 3-channel
image.
#### Page extraction
@ -536,7 +693,7 @@ image.
"thetha" : [10, -10],
"continue_training": false,
"index_start" : 0,
"dir_of_start_model" : " ",
"dir_of_start_model" : " ",
"weighted_loss": false,
"is_loss_soft_dice": false,
"data_is_provided": false,
@ -546,10 +703,11 @@ image.
}
```
For page segmentation (or printspace or border segmentation), the model needs to view the input image in its entirety,
hence the patches parameter should be set to false.
For page segmentation (or printspace or border segmentation), the model needs to view the input image in its
entirety,hence the patches parameter should be set to false.
#### layout segmentation
An example config json file for layout segmentation with 5 classes (including background) can be like this:
```yaml
@ -593,7 +751,7 @@ An example config json file for layout segmentation with 5 classes (including ba
"thetha" : [10, -10],
"continue_training": false,
"index_start" : 0,
"dir_of_start_model" : " ",
"dir_of_start_model" : " ",
"weighted_loss": false,
"is_loss_soft_dice": false,
"data_is_provided": false,
@ -605,28 +763,42 @@ An example config json file for layout segmentation with 5 classes (including ba
## Inference with the trained model
### classification
For conducting inference with a trained model, you simply need to execute the following command line, specifying the
For conducting inference with a trained model, you simply need to execute the following command line, specifying the
directory of the model and the image on which to perform inference:
