- pass on probabilities from predicted class everywhere
- rename `confidence_matrix` → `confidence_regions` / `regions_confidence`
- rename `get_textregion_confidences()` → `get_region_confidences()`
- add same for tables, textlines and regionsfl (full layout model)
- aggregate per-region confidence lists for image, table, drop-capital,
left marginal and right marginal regions
- add in writer
- simplify/re-indent some
- try to replace more number literals with class label identifiers
- re-introduce boosting `heading` thresholding broken
when refactoring (light version and do_prediction)
- also return confidence for full layout prediction
1. use connected component analysis to get unique segments
in early prediction result
2. for each drop-capital segment in full prediction result,
find matching early segment
3. when they have high overlap, assign drop-capital label
to the entire early segment
- rename `putt_bb_of_drop_capitals_of_model_in_patches_in_layout`
→ `fill_bb_of_drop_capitals`
- also allow image (besides text) label in early layout prediction
result when checking if entire bbox can be filled (as opposed to
just drop-capital | image | background mask)
- simplify
fix bug where in non-full mode, the wrong class label was assumed
for separator regions (3 in non- vs 6 in full layout mode):
- pass in separator mask instead of full segmentation map
- rename for clarity:
- `regions_without_separators` → `text_mask` (alread binary)
- `regions_with_separators` → `sep_mask` (now just binary)
(thresholding and decoding with artificial boundary class can
overwrite existing column separators, which in turn can contribute
to missing column boundaries; this prioritises seps over boundaries,
which does not impair separation of instances, as seps will separate
text/image/etc instances just as well as artificial boundaries)
When 338c4a0e wrapped all prediction models for automatic
image size adaptation in CUDA,
- tiling (`_patched`) was indeed faster
- whole (`_resized`) was actually slower
But CUDA-based tiling also increases GPU memory requirements
a lot. And with the new parallel subprocess predictors, Numpy-
based tiling is not necessarily slower anymore.
(avoid strange image handling short-cut, which uses
early cropped image used for column classification
instead of normal image in 1/2-column cases;
fixes accuracy issues of region_1_2 model on these images)
(as follow-up to ec08004f:)
- create log queues and QueueListener separately for each job
- receive job logs sequentially
- drop log filter mechanism (prefixing log messages by file name)
- also count ratio of successful jobs
allow setting device specifier to load models into
either
- CPU or single GPU0, GPU1 etc
- per-model patterns, e.g. col*:CPU,page:GPU0,*:GPU1
pass through as kwargs until `ModelZoo.load_models()` setup up TF
- Eynollah: instead of one `Predictor` instance as stand-in for
entire `ModelZoo`, keep the latter but have each model in `_loaded`
dict become an independent predictor instance
- `ModelZoo.load_models()`: instantiate `Predictor`s for each
`model_category` and then call `Predictor.load_model()` on them
- `Predictor.load_model()`: set args/kwargs for `ModelZoo.load_model()`,
then spawn subprocess via `.start()`, which first enters `setup()`...
- `Predictor.setup()`: call `ModelZoo.load_model()` instead of (plural)
`.load_models()`; save to `self.model` instead of `self.model_zoo`
- `ModelZoo.load_model()`: move _all_ CUDA configuration and
TF/Keras-specific module initialization here (to be used only by
predictor subprocess)
- `Predictor`: drop stand-in `SingleModelPredictor` retrieved by `get()`;
directly provide `predict()` and `output_shape` via `self.call()`
- `Predictor`: drop `model` arg from all queues - now implicit; use
`self.name` for model name in messages
- `Predictor`: no need for requeuing other tasks (only same model now)
- `Predictor`: reduce rebatching batch sizes due to increased VRAM footprint
- `Eynollah.setup_models()`: set up loading `_patched` / `_resized`
here instead of during `ModelZoo.load_model()`
- `ModelZoo.load_models()`: for resized/patched models, call
`Predictor.load_model()` with kwarg instead of resp. model name suffix
- `ModelZoo.load_model()`: expect boolean kwargs `patched/resized`
for `wrap_layout_model_patched/resized` model wrappers, respectively
- depending on model type (i.e. size), configure target
batch sizes
- after receiving a prediction task for some model,
look up target batch size, then try to retrieve arrays
from follow-up tasks for the same model on the task queue;
stop when either no tasks are immediately available or
when the combined batch size (input batch size * number of tasks)
reaches the target
- push back tasks for other models to the queue
- rebatch: read all shared arrays, concatenate them along axis 0,
map respective job ids they came from
- predict on new (possibly larger) batch
- split result along axis 0 into number of jobs
- send each result along with its jobid to task queue
- set up a Queue and QueueListener along with ProcessPoolExecutor,
delegating messages from the queue to all handlers
- in forked subprocesses, instead of just inheriting handlers,
replace them with a single QueueHandler, and make sure
log messages get prefixes by the respective job id (img_filename)
so concurrent messages will still be readable
- in the predictor, make sure to pass on the log level to the
spawned subprocess, too
When 338c4a0e wrapped all prediction models for automatic
image size adaptation in CUDA,
- tiling (`_patched`) was indeed faster
- whole (`_resized`) was actually slower
So this reverts the latter part.
- reintroduce ProcessPoolExecutor
(previously for parallel deskewing within pages)
- wrap Eynollah instance into global, so (with forking)
serialization can be avoided – same pattern as in core ocrd.Processor
- move timing/logging into `run_single()`, respectively
- `cache_images()`: only return an image dict (plus extra keys
for file name stem and dpi) - don't set any attributes
- `imread()`: just take from passed image dict, also add `binary` key
- `resize_and_enhance_image_with_column_classifier()`:
* `imread()` from image dict
* set `img_bin` key for binarization result if `input_binary`
* instead of `image_page_org_size` / `page_coord` attributes,
set `img_page` / `coord_page` in image dict
* instead of retval, set `img_res` in image dict
* also set `scale_x` and `scale_y` in image dict, resp.
* simplify
- `resize_image_with_column_classifier()`:
* `imread()` from image dict
* (as in `resize_and_enhance_with_column_classifier`:)
call `calculate_width_height_by_columns_1_2` if `num_col` is
1 or 2 here
* instead of retval, set `img_res` in image dict
* also set `scale_x` and `scale_y` in image dict, resp.
* simplify
- `calculate_width_height_by_columns*()`: simplify, get confidence of
num_col instead of entire array
- `extract_page()`: read `img_res` from image dict; simplify
- `early_page_for_num_of_column_classification()`:
`imread()` from image dict; simplify
- `textline_contours()`: no need for `num_col_classifier` here
- `run_textline()`: no need for `num_col_classifier` here
- `get_regions_light_v()` → `get_regions()`:
* read `img_res` from image dict
* get shapes via `img` from image dict instead of `image_org` attr
* use `img_page` / `coord_page` from image dict instead of attrs
* avoid unnecessary 3-channel arrays
* simplify
- `get_tables_from_model()`: no need for `num_col_classifier` here
- `run_graphics_and_columns_light()` → `run_graphics_and_columns()`:
* pass through image dict instead of `img_bin` (which really was `img_res`)
* simplify
- `run_graphics_and_columns_without_layout()`:
* pass through image dict instead of `img_bin` (which really was `img_res`)
* simplify
- `run_enhancement()`: pass through image dict
- `get_image_and_sclaes*()`: drop
- `run_boxes_full_layout()`:
* pass `image_page` instead of `img_bin` (which really was `image_page`)
* simplify
- `run()`:
* instantiate plotter outside of loop, and independent of img files
* move writer instantiation and overwrite checks into `run_single()`
* add try/catch for `run_single()` w/ logging
- `reset_file_name_dir`: drop
- `run_single()`:
* add some args/kwargs from `run()`
* call `cache_images()` (reading image dict) here
* instantiate writer here instead of (reused) attr in `run()`
* set `scale_x` / `scale_y` in writer from image dict once known
(i.e. after `run_enhancement()`)
* don't return anything, but write PAGE result here
- `check_any_text_region_in_model_one_is_main_or_header_light()` →
`split_textregion_main_vs_header()`
- plotter:
* pass `name` (file stem) from image dict to all methods
* for `write_images_into_directory()`: also `scale_x` and `scale_y`
from image dict
- writer:
* init with width/height
- ocrd processor:
* adapt (just `run_single()` call)
* drop `max_workers=1` restriction (can now run fully parallel)
- `get_textregion_contours_in_org_image_light()` →
`get_textregion_confidences()`:
* take shape from confmat directly instead of extra array
* simplify
When `num_col_classifier` predicted result gets bypassed
by heuristic result from `find_num_col()` (because prediction
had too little confidence or `calculate_width_height_by_columns()`
would have become too large), do not increment `num_col` further
(already 1 more than colseps).
- rename `terminate` → `stopped`
- call `terminate()` from superclass during shutdown
- del `self.model_zoo` in the parent process after spawn,
and in the child during shutdown
- new class `Predictor(multiprocessing.Process)` as stand-in
for EynollahModelZoo:
* calling `load_models()` starts the subprocess (and has
`.model_zoo.load_models()` run internally)
* calling `get()` yields a stand-in that supports `.predict()`,
which actually communicates with the singleton subprocess
via task and result queues, sharing Numpy arrays via SHM
* calling `predict()` with an empty dict (instead of an image)
merely retrieves the respective model's output shapes (cached)
* shared memory objects for arrays are cleared as soon as possible
* log messages are piped through QueueHandler / QueueListener
* exceptions are passed through the queues, and raised afterwards
- move all TF initialization to the predictor
(to avoid back and forth between CPU and GPU memory when looping
over image patches)
- `patch_encoder`: define `Model` subclasses which take an existing
(layout segmentation) model in the constructor, and define a new
`call()` using the existing model in a GPU-only `tf.function`:
* `wrap_layout_model_resized`: just `tf.image.resize()` from
input image to model size, then predict, then resize back
* `wrap_layout_model_patched`: ditto if smaller than model size;
otherwise use `tf.image.extract_patches` for patching in a
sliding-window approach, then predict patches one by one, then
`tf.scatter_nd` to reconstruct to image size
- when compiling `tf.function` graph, make sure to use input signature
with variable image size, but avoid retracing each new size sample
- in `EynollahModelZoo.load_model` for relevant model types,
also wrap the loaded model
* by `wrap_layout_model_resized` under model name + `_resized`
* by `wrap_layout_model_patched` under model name + `_patched`
- introduce `do_prediction_new_concept_autosize`,
replacing `do_prediction/_new_concept`,
but using passed model's `predict` directly without
resizing or tiling to model size
- instead of `do_prediction/_new_concept(True, ...)`,
now call `do_prediction_new_concept_autosize`,
but with `_patched` appended to model name
- instead of `do_prediction/_new_concept(False, ...)`,
now call `do_prediction_new_concept_autosize`,
but with `_resized` appended to model name
