diff --git a/README.md b/README.md index 07253ee..4d430bc 100644 --- a/README.md +++ b/README.md @@ -43,10 +43,10 @@ Alternatively, you can run `make install` or `make install-dev` for editable ins ## Models Pre-trained models can be downloaded from [qurator-data.de](https://qurator-data.de/eynollah/) or [huggingface](https://huggingface.co/SBB?search_models=eynollah). +For documentation on methods and models, have a look at [`models.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/models.md). + ## Train -🚧 **Work in progress** - -In case you want to train your own model, have a look at [`sbb_pixelwise_segmentation`](https://github.com/qurator-spk/sbb_pixelwise_segmentation). +In case you want to train your own model with Eynollah, have a look at [`train.md`](https://github.com/qurator-spk/eynollah/tree/main/docs/train.md). ## Usage The command-line interface can be called like this: @@ -84,7 +84,8 @@ The best output quality is produced when RGB images are used as input rather tha #### Use as OCR-D processor -Eynollah ships with a CLI interface to be used as [OCR-D](https://ocr-d.de) [processor](https://ocr-d.de/en/spec/cli). +Eynollah ships with a CLI interface to be used as [OCR-D](https://ocr-d.de) [processor](https://ocr-d.de/en/spec/cli), +formally described in [`ocrd-tool.json`](https://github.com/qurator-spk/eynollah/tree/main/src/eynollah/ocrd-tool.json). In this case, the source image file group with (preferably) RGB images should be used as input like this: diff --git a/docs/models.md b/docs/models.md new file mode 100644 index 0000000..ac563b0 --- /dev/null +++ b/docs/models.md @@ -0,0 +1,147 @@ +# Models documentation +This suite of 14 models presents a document layout analysis (DLA) system for historical documents implemented by +pixel-wise segmentation using a combination of a ResNet50 encoder with various U-Net decoders. In addition, heuristic +methods are applied to detect marginals and to determine the reading order of text regions. + +The detection and classification of multiple classes of layout elements such as headings, images, tables etc. as part of +DLA is required in order to extract and process them in subsequent steps. Altogether, the combination of image +detection, classification and segmentation on the wide variety that can be found in over 400 years of printed cultural +heritage makes this a very challenging task. Deep learning models are complemented with heuristics for the detection of +text lines, marginals, and reading order. Furthermore, an optional image enhancement step was added in case of documents +that either have insufficient pixel density and/or require scaling. Also, a column classifier for the analysis of +multi-column documents was added. With these additions, DLA performance was improved, and a high accuracy in the +prediction of the reading order is accomplished. + +Two Arabic/Persian terms form the name of the model suite: عين الله, which can be transcribed as "ain'allah" or +"eynollah"; it translates into English as "God's Eye" -- it sees (nearly) everything on the document image. + +See the flowchart below for the different stages and how they interact: + +![](https://user-images.githubusercontent.com/952378/100619946-1936f680-331e-11eb-9297-6e8b4cab3c16.png) + + +## Models + +### Image enhancement +Model card: [Image Enhancement](https://huggingface.co/SBB/eynollah-enhancement) + +This model addresses image resolution, specifically targeting documents with suboptimal resolution. In instances where +the detection of document layout exhibits inadequate performance, the proposed enhancement aims to significantly improve +the quality and clarity of the images, thus facilitating enhanced visual interpretation and analysis. + +### Page extraction / border detection +Model card: [Page Extraction/Border Detection](https://huggingface.co/SBB/eynollah-page-extraction) + +A problem that can negatively affect OCR are black margins around a page caused by document scanning. A deep learning +model helps to crop to the page borders by using a pixel-wise segmentation method. + +### Column classification +Model card: [Column Classification](https://huggingface.co/SBB/eynollah-column-classifier) + +This model is a trained classifier that recognizes the number of columns in a document by use of a training set with +manual classification of all documents into six classes with either one, two, three, four, five, or six and more columns +respectively. + +### Binarization +Model card: [Binarization](https://huggingface.co/SBB/eynollah-binarization) + +This model is designed to tackle the intricate task of document image binarization, which involves segmentation of the +image into white and black pixels. This process significantly contributes to the overall performance of the layout +models, particularly in scenarios where the documents are degraded or exhibit subpar quality. The robust binarization +capability of the model enables improved accuracy and reliability in subsequent layout analysis, thereby facilitating +enhanced document understanding and interpretation. + +### Main region detection +Model card: [Main Region Detection](https://huggingface.co/SBB/eynollah-main-regions) + +This model has employed a different set of labels, including an artificial class specifically designed to encompass the +text regions. The inclusion of this artificial class facilitates easier isolation of text regions by the model. This +approach grants the advantage of training the model using downscaled images, which in turn leads to faster predictions +during the inference phase. By incorporating this methodology, improved efficiency is achieved without compromising the +model's ability to accurately identify and classify text regions within documents. + +### Main region detection (with scaling augmentation) +Model card: [Main Region Detection (with scaling augmentation)](https://huggingface.co/SBB/eynollah-main-regions-aug-scaling) + +Utilizing scaling augmentation, this model leverages the capability to effectively segment elements of extremely high or +low scales within documents. By harnessing this technique, the tool gains a significant advantage in accurately +categorizing and isolating such elements, thereby enhancing its overall performance and enabling precise analysis of +documents with varying scale characteristics. + +### Main region detection (with rotation augmentation) +Model card: [Main Region Detection (with rotation augmentation)](https://huggingface.co/SBB/eynollah-main-regions-aug-rotation) + +This model takes advantage of rotation augmentation. This helps the tool to segment the vertical text regions in a +robust way. + +### Main region detection (ensembled) +Model card: [Main Region Detection (ensembled)](https://huggingface.co/SBB/eynollah-main-regions-ensembled) + +The robustness of this model is attained through an ensembling technique that combines the weights from various epochs. +By employing this approach, the model achieves a high level of resilience and stability, effectively leveraging the +strengths of multiple epochs to enhance its overall performance and deliver consistent and reliable results. + +### Full region detection (1,2-column documents) +Model card: [Full Region Detection (1,2-column documents)](https://huggingface.co/SBB/eynollah-full-regions-1column) + +This model deals with documents comprising of one and two columns. + +### Full region detection (3,n-column documents) +Model card: [Full Region Detection (3,n-column documents)](https://huggingface.co/SBB/eynollah-full-regions-3pluscolumn) + +This model is responsible for detecting headers and drop capitals in documents with three or more columns. + +### Textline detection +Model card: [Textline Detection](https://huggingface.co/SBB/eynollah-textline) + +The method for textline detection combines deep learning and heuristics. In the deep learning part, an image-to-image +model performs binary segmentation of the document into the classes textline vs. background. In the heuristics part, +bounding boxes or contours are derived from binary segmentation. + +Skewed documents can heavily affect textline detection accuracy, so robust deskewing is needed. But detecting textlines +with rectangle bounding boxes cannot deal with partially curved textlines. To address this, a functionality +specifically for documents with curved textlines was included. After finding the contour of a text region and its +corresponding textline segmentation, the text region is cut into smaller vertical straps. For each strap, its textline +segmentation is first deskewed and then the textlines are separated with the same heuristic method as for finding +textline bounding boxes. Later, the strap is rotated back into its original orientation. + +### Textline detection (light) +Model card: [Textline Detection Light (simpler but faster method)](https://huggingface.co/SBB/eynollah-textline_light) + +The method for textline detection combines deep learning and heuristics. In the deep learning part, an image-to-image +model performs binary segmentation of the document into the classes textline vs. background. In the heuristics part, +bounding boxes or contours are derived from binary segmentation. + +In the context of this textline model, a distinct labeling approach has been employed to ensure accurate predictions. +Specifically, an artificial bounding class has been incorporated alongside the textline classes. This strategic +inclusion effectively prevents any spurious connections between adjacent textlines during the prediction phase, thereby +enhancing the model's ability to accurately identify and delineate individual textlines within documents. This model +eliminates the need for additional heuristics in extracting textline contours. + +### Table detection +Model card: [Table Detection](https://huggingface.co/SBB/eynollah-tables) + +The objective of this model is to perform table segmentation in historical document images. Due to the pixel-wise +segmentation approach employed and the presence of traditional tables predominantly composed of text, the detection of +tables required the incorporation of heuristics to achieve reasonable performance. These heuristics were necessary to +effectively identify and delineate tables within the historical document images, ensuring accurate segmentation and +enabling subsequent analysis and interpretation. + +### Image detection +Model card: [Image Detection](https://huggingface.co/SBB/eynollah-image-extraction) + +This model is used for the task of illustration detection only. + +### Reading order detection +Model card: [Reading Order Detection]() + +TODO + +## Heuristic methods +Additionally, some heuristic methods are employed to further improve the model predictions: +* After border detection, the largest contour is determined by a bounding box, and the image cropped to these coordinates. +* For text region detection, the image is scaled up to make it easier for the model to detect background space between text regions. +* A minimum area is defined for text regions in relation to the overall image dimensions, so that very small regions that are noise can be filtered out. +* Deskewing is applied on the text region level (due to regions having different degrees of skew) in order to improve the textline segmentation result. +* After deskewing, a calculation of the pixel distribution on the X-axis allows the separation of textlines (foreground) and background pixels. +* Finally, using the derived coordinates, bounding boxes are determined for each textline. diff --git a/docs/train.md b/docs/train.md new file mode 100644 index 0000000..9f44a63 --- /dev/null +++ b/docs/train.md @@ -0,0 +1,632 @@ +# Training documentation +This aims to assist users in preparing training datasets, training models, and performing inference with trained models. +We cover various use cases including pixel-wise segmentation, image classification, image enhancement, and machine-based +reading order detection. For each use case, we provide guidance on how to generate the corresponding training dataset. + +The following three tasks can all be accomplished using the code in the +[`train`](https://github.com/qurator-spk/sbb_pixelwise_segmentation/tree/unifying-training-models) directory: + +* generate training dataset +* train a model +* inference with the trained model + +## Generate training dataset +The script `generate_gt_for_training.py` is used for generating training datasets. As the results of the following +command demonstrates, the dataset generator provides three different commands: + +`python generate_gt_for_training.py --help` + +These three commands are: + +* image-enhancement +* machine-based-reading-order +* pagexml2label + +### image-enhancement +Generating a training dataset for image enhancement is quite straightforward. All that is needed is a set of +high-resolution images. The training dataset can then be generated using the following command: + +`python generate_gt_for_training.py image-enhancement -dis "dir of high resolution images" -dois "dir where degraded +images will be written" -dols "dir where the corresponding high resolution image will be written as label" -scs +"degrading scales json file"` + +The scales JSON file is a dictionary with a key named 'scales' and values representing scales smaller than 1. Images are +downscaled based on these scales and then upscaled again to their original size. This process causes the images to lose +resolution at different scales. The degraded images are used as input images, and the original high-resolution images +serve as labels. The enhancement model can be trained with this generated dataset. The scales JSON file looks like this: + +```yaml +{ + "scales": [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9] +} +``` + +### machine-based-reading-order +For machine-based reading order, we aim to determine the reading priority between two sets of text regions. The model's +input is a three-channel image: the first and last channels contain information about each of the two text regions, +while the middle channel encodes prominent layout elements necessary for reading order, such as separators and headers. +To generate the training dataset, our script requires a page XML file that specifies the image layout with the correct +reading order. + +For output images, it is necessary to specify the width and height. Additionally, a minimum text region size can be set +to filter out regions smaller than this minimum size. This minimum size is defined as the ratio of the text region area +to the image area, with a default value of zero. To run the dataset generator, use the following command: + +`python generate_gt_for_training.py machine-based-reading-order -dx "dir of GT xml files" -domi "dir where output images +will be written" -docl "dir where the labels will be written" -ih "height" -iw "width" -min "min area ratio"` + +### pagexml2label +pagexml2label is designed to generate labels from GT page XML files for various pixel-wise segmentation use cases, +including 'layout,' 'textline,' 'printspace,' 'glyph,' and 'word' segmentation. +To train a pixel-wise segmentation model, we require images along with their corresponding labels. Our training script +expects a PNG image where each pixel corresponds to a label, represented by an integer. The background is always labeled +as zero, while other elements are assigned different integers. For instance, if we have ground truth data with four +elements including the background, the classes would be labeled as 0, 1, 2, and 3 respectively. + +In binary segmentation scenarios such as textline or page extraction, the background is encoded as 0, and the desired +element is automatically encoded as 1 in the PNG label. + +To specify the desired use case and the elements to be extracted in the PNG labels, a custom JSON file can be passed. +For example, in the case of 'textline' detection, the JSON file would resemble this: + +```yaml +{ +"use_case": "textline" +} +``` + +In the case of layout segmentation a custom config json file can look like this: + +```yaml +{ +"use_case": "layout", +"textregions":{"rest_as_paragraph":1 , "drop-capital": 1, "header":2, "heading":2, "marginalia":3}, +"imageregion":4, +"separatorregion":5, +"graphicregions" :{"rest_as_decoration":6 ,"stamp":7} +} +``` + +A possible custom config json file for layout segmentation where the "printspace" is a class: + +```yaml +{ +"use_case": "layout", +"textregions":{"rest_as_paragraph":1 , "drop-capital": 1, "header":2, "heading":2, "marginalia":3}, +"imageregion":4, +"separatorregion":5, +"graphicregions" :{"rest_as_decoration":6 ,"stamp":7} +"printspace_as_class_in_layout" : 8 +} +``` + +For the layout use case, it is beneficial to first understand the structure of the page XML file and its elements. +In a given image, the annotations of elements are recorded in a page XML file, including their contours and classes. +For an image document, the known regions are 'textregion', 'separatorregion', 'imageregion', 'graphicregion', +'noiseregion', and 'tableregion'. + +Text regions and graphic regions also have their own specific types. The known types for text regions are 'paragraph', +'header', 'heading', 'marginalia', 'drop-capital', 'footnote', 'footnote-continued', 'signature-mark', 'page-number', +and 'catch-word'. The known types for graphic regions are 'handwritten-annotation', 'decoration', 'stamp', and +'signature'. +Since we don't know all types of text and graphic regions, unknown cases can arise. To handle these, we have defined +two additional types, "rest_as_paragraph" and "rest_as_decoration", to ensure that no unknown types are missed. +This way, users can extract all known types from the labels and be confident that no unknown types are overlooked. + +In the custom JSON file shown above, "header" and "heading" are extracted as the same class, while "marginalia" is shown +as a different class. All other text region types, including "drop-capital," are grouped into the same class. For the +graphic region, "stamp" has its own class, while all other types are classified together. "Image region" and "separator +region" are also present in the label. However, other regions like "noise region" and "table region" will not be +included in the label PNG file, even if they have information in the page XML files, as we chose not to include them. + +`python generate_gt_for_training.py pagexml2label -dx "dir of GT xml files" -do "dir where output label png files will +be written" -cfg "custom config json file" -to "output type which has 2d and 3d. 2d is used for training and 3d is just +to visualise the labels" "` + +We have also defined an artificial class that can be added to the boundary of text region types or text lines. This key +is called "artificial_class_on_boundary." If users want to apply this to certain text regions in the layout use case, +the example JSON config file should look like this: + +```yaml +{ + "use_case": "layout", + "textregions": { + "paragraph": 1, + "drop-capital": 1, + "header": 2, + "heading": 2, + "marginalia": 3 + }, + "imageregion": 4, + "separatorregion": 5, + "graphicregions": { + "rest_as_decoration": 6 + }, + "artificial_class_on_boundary": ["paragraph", "header", "heading", "marginalia"], + "artificial_class_label": 7 +} +``` + +This implies that the artificial class label, denoted by 7, will be present on PNG files and will only be added to the +elements labeled as "paragraph," "header," "heading," and "marginalia." + +For "textline", "word", and "glyph", the artificial class on the boundaries will be activated only if the +"artificial_class_label" key is specified in the config file. Its value should be set as 2 since these elements +represent binary cases. For example, if the background and textline are denoted as 0 and 1 respectively, then the +artificial class should be assigned the value 2. The example JSON config file should look like this for "textline" use +case: + +```yaml +{ + "use_case": "textline", + "artificial_class_label": 2 +} +``` + +If the coordinates of "PrintSpace" or "Border" are present in the page XML ground truth files, and the user wishes to +crop only the print space area, this can be achieved by activating the "-ps" argument. However, it should be noted that +in this scenario, since cropping will be applied to the label files, the directory of the original images must be +provided to ensure that they are cropped in sync with the labels. This ensures that the correct images and labels +required for training are obtained. The command should resemble the following: + +`python generate_gt_for_training.py pagexml2label -dx "dir of GT xml files" -do "dir where output label png files will +be written" -cfg "custom config json file" -to "output type which has 2d and 3d. 2d is used for training and 3d is just +to visualise the labels" -ps -di "dir where the org images are located" -doi "dir where the cropped output images will +be written" ` + +## Train a model +### classification + +For the classification use case, we haven't provided a ground truth generator, as it's unnecessary. For classification, +all we require is a training directory with subdirectories, each containing images of its respective classes. We need +separate directories for training and evaluation, and the class names (subdirectories) must be consistent across both +directories. Additionally, the class names should be specified in the config JSON file, as shown in the following +example. If, for instance, we aim to classify "apple" and "orange," with a total of 2 classes, the +"classification_classes_name" key in the config file should appear as follows: + +```yaml +{ + "backbone_type" : "nontransformer", + "task": "classification", + "n_classes" : 2, + "n_epochs" : 10, + "input_height" : 448, + "input_width" : 448, + "weight_decay" : 1e-6, + "n_batch" : 4, + "learning_rate": 1e-4, + "f1_threshold_classification": 0.8, + "pretraining" : true, + "classification_classes_name" : {"0":"apple", "1":"orange"}, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +The "dir_train" should be like this: + +``` +. +└── train # train directory + ├── apple # directory of images for apple class + └── orange # directory of images for orange class +``` + +And the "dir_eval" the same structure as train directory: + +``` +. +└── eval # evaluation directory + ├── apple # directory of images for apple class + └── orange # directory of images for orange class + +``` + +The classification model can be trained using the following command line: + +`python train.py with config_classification.json` + +As evident in the example JSON file above, for classification, we utilize a "f1_threshold_classification" parameter. +This parameter is employed to gather all models with an evaluation f1 score surpassing this threshold. Subsequently, +an ensemble of these model weights is executed, and a model is saved in the output directory as "model_ens_avg". +Additionally, the weight of the best model based on the evaluation f1 score is saved as "model_best". + +### reading order +An example config json file for machine based reading order should be like this: + +```yaml +{ + "backbone_type" : "nontransformer", + "task": "reading_order", + "n_classes" : 1, + "n_epochs" : 5, + "input_height" : 672, + "input_width" : 448, + "weight_decay" : 1e-6, + "n_batch" : 4, + "learning_rate": 1e-4, + "pretraining" : true, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +The "dir_train" should be like this: + +``` +. +└── train # train directory + ├── images # directory of images + └── labels # directory of labels +``` + +And the "dir_eval" the same structure as train directory: + +``` +. +└── eval # evaluation directory + ├── images # directory of images + └── labels # directory of labels +``` + +The classification model can be trained like the classification case command line. + +### Segmentation (Textline, Binarization, Page extraction and layout) and enhancement + +#### Parameter configuration for segmentation or enhancement usecases +The following parameter configuration can be applied to all segmentation use cases and enhancements. The augmentation, +its sub-parameters, and continued training are defined only for segmentation use cases and enhancements, not for +classification and machine-based reading order, as you can see in their example config files. + +* backbone_type: For segmentation tasks (such as text line, binarization, and layout detection) and enhancement, we +* offer two backbone options: a "nontransformer" and a "transformer" backbone. For the "transformer" backbone, we first +* apply a CNN followed by a transformer. In contrast, the "nontransformer" backbone utilizes only a CNN ResNet-50. +* task : The task parameter can have values such as "segmentation", "enhancement", "classification", and "reading_order". +* patches: If you want to break input images into smaller patches (input size of the model) you need to set this +* parameter to ``true``. In the case that the model should see the image once, like page extraction, patches should be +* set to ``false``. +* n_batch: Number of batches at each iteration. +* n_classes: Number of classes. In the case of binary classification this should be 2. In the case of reading_order it +* should set to 1. And for the case of layout detection just the unique number of classes should be given. +* n_epochs: Number of epochs. +* input_height: This indicates the height of model's input. +* input_width: This indicates the width of model's input. +* weight_decay: Weight decay of l2 regularization of model layers. +* pretraining: Set to ``true`` to load pretrained weights of ResNet50 encoder. The downloaded weights should be saved +* in a folder named "pretrained_model" in the same directory of "train.py" script. +* augmentation: If you want to apply any kind of augmentation this parameter should first set to ``true``. +* flip_aug: If ``true``, different types of filp will be applied on image. Type of flips is given with "flip_index" parameter. +* blur_aug: If ``true``, different types of blurring will be applied on image. Type of blurrings is given with "blur_k" parameter. +* scaling: If ``true``, scaling will be applied on image. Scale of scaling is given with "scales" parameter. +* degrading: If ``true``, degrading will be applied to the image. The amount of degrading is defined with "degrade_scales" parameter. +* brightening: If ``true``, brightening will be applied to the image. The amount of brightening is defined with "brightness" parameter. +* rotation_not_90: If ``true``, rotation (not 90 degree) will be applied on image. Rotation angles are given with "thetha" parameter. +* rotation: If ``true``, 90 degree rotation will be applied on image. +* binarization: If ``true``,Otsu thresholding will be applied to augment the input data with binarized images. +* scaling_bluring: If ``true``, combination of scaling and blurring will be applied on image. +* scaling_binarization: If ``true``, combination of scaling and binarization will be applied on image. +* scaling_flip: If ``true``, combination of scaling and flip will be applied on image. +* flip_index: Type of flips. +* blur_k: Type of blurrings. +* scales: Scales of scaling. +* brightness: The amount of brightenings. +* thetha: Rotation angles. +* degrade_scales: The amount of degradings. +* continue_training: If ``true``, it means that you have already trained a model and you would like to continue the training. So it is needed to provide the dir of trained model with "dir_of_start_model" and index for naming the models. For example if you have already trained for 3 epochs then your last index is 2 and if you want to continue from model_1.h5, you can set ``index_start`` to 3 to start naming model with index 3. +* weighted_loss: If ``true``, this means that you want to apply weighted categorical_crossentropy as loss fucntion. Be carefull if you set to ``true``the parameter "is_loss_soft_dice" should be ``false`` +* data_is_provided: If you have already provided the input data you can set this to ``true``. Be sure that the train and eval data are in "dir_output". Since when once we provide training data we resize and augment them and then we write them in sub-directories train and eval in "dir_output". +* dir_train: This is the directory of "images" and "labels" (dir_train should include two subdirectories with names of images and labels ) for raw images and labels. Namely they are not prepared (not resized and not augmented) yet for training the model. When we run this tool these raw data will be transformed to suitable size needed for the model and they will be written in "dir_output" in train and eval directories. Each of train and eval include "images" and "labels" sub-directories. +* index_start: Starting index for saved models in the case that "continue_training" is ``true``. +* dir_of_start_model: Directory containing pretrained model to continue training the model in the case that "continue_training" is ``true``. +* transformer_num_patches_xy: Number of patches for vision transformer in x and y direction respectively. +* transformer_patchsize_x: Patch size of vision transformer patches in x direction. +* transformer_patchsize_y: Patch size of vision transformer patches in y direction. +* transformer_projection_dim: Transformer projection dimension. Default value is 64. +* transformer_mlp_head_units: Transformer Multilayer Perceptron (MLP) head units. Default value is [128, 64]. +* transformer_layers: transformer layers. Default value is 8. +* transformer_num_heads: Transformer number of heads. Default value is 4. +* transformer_cnn_first: We have two types of vision transformers. In one type, a CNN is applied first, followed by a transformer. In the other type, this order is reversed. If transformer_cnn_first is true, it means the CNN will be applied before the transformer. Default value is true. + +In the case of segmentation and enhancement the train and evaluation directory should be as following. + +The "dir_train" should be like this: + +``` +. +└── train # train directory + ├── images # directory of images + └── labels # directory of labels +``` + +And the "dir_eval" the same structure as train directory: + +``` +. +└── eval # evaluation directory + ├── images # directory of images + └── labels # directory of labels +``` + +After configuring the JSON file for segmentation or enhancement, training can be initiated by running the following +command, similar to the process for classification and reading order: + +`python train.py with config_classification.json` + +#### Binarization +An example config json file for binarization can be like this: + +```yaml +{ + "backbone_type" : "transformer", + "task": "binarization", + "n_classes" : 2, + "n_epochs" : 4, + "input_height" : 224, + "input_width" : 672, + "weight_decay" : 1e-6, + "n_batch" : 1, + "learning_rate": 1e-4, + "patches" : true, + "pretraining" : true, + "augmentation" : true, + "flip_aug" : false, + "blur_aug" : false, + "scaling" : true, + "degrading": false, + "brightening": false, + "binarization" : false, + "scaling_bluring" : false, + "scaling_binarization" : false, + "scaling_flip" : false, + "rotation": false, + "rotation_not_90": false, + "transformer_num_patches_xy": [7, 7], + "transformer_patchsize_x": 3, + "transformer_patchsize_y": 1, + "transformer_projection_dim": 192, + "transformer_mlp_head_units": [128, 64], + "transformer_layers": 8, + "transformer_num_heads": 4, + "transformer_cnn_first": true, + "blur_k" : ["blur","guass","median"], + "scales" : [0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.4], + "brightness" : [1.3, 1.5, 1.7, 2], + "degrade_scales" : [0.2, 0.4], + "flip_index" : [0, 1, -1], + "thetha" : [10, -10], + "continue_training": false, + "index_start" : 0, + "dir_of_start_model" : " ", + "weighted_loss": false, + "is_loss_soft_dice": false, + "data_is_provided": false, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +#### Textline + +```yaml +{ + "backbone_type" : "nontransformer", + "task": "segmentation", + "n_classes" : 2, + "n_epochs" : 4, + "input_height" : 448, + "input_width" : 224, + "weight_decay" : 1e-6, + "n_batch" : 1, + "learning_rate": 1e-4, + "patches" : true, + "pretraining" : true, + "augmentation" : true, + "flip_aug" : false, + "blur_aug" : false, + "scaling" : true, + "degrading": false, + "brightening": false, + "binarization" : false, + "scaling_bluring" : false, + "scaling_binarization" : false, + "scaling_flip" : false, + "rotation": false, + "rotation_not_90": false, + "blur_k" : ["blur","guass","median"], + "scales" : [0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.4], + "brightness" : [1.3, 1.5, 1.7, 2], + "degrade_scales" : [0.2, 0.4], + "flip_index" : [0, 1, -1], + "thetha" : [10, -10], + "continue_training": false, + "index_start" : 0, + "dir_of_start_model" : " ", + "weighted_loss": false, + "is_loss_soft_dice": false, + "data_is_provided": false, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +#### Enhancement + +```yaml +{ + "backbone_type" : "nontransformer", + "task": "enhancement", + "n_classes" : 3, + "n_epochs" : 4, + "input_height" : 448, + "input_width" : 224, + "weight_decay" : 1e-6, + "n_batch" : 4, + "learning_rate": 1e-4, + "patches" : true, + "pretraining" : true, + "augmentation" : true, + "flip_aug" : false, + "blur_aug" : false, + "scaling" : true, + "degrading": false, + "brightening": false, + "binarization" : false, + "scaling_bluring" : false, + "scaling_binarization" : false, + "scaling_flip" : false, + "rotation": false, + "rotation_not_90": false, + "blur_k" : ["blur","guass","median"], + "scales" : [0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.4], + "brightness" : [1.3, 1.5, 1.7, 2], + "degrade_scales" : [0.2, 0.4], + "flip_index" : [0, 1, -1], + "thetha" : [10, -10], + "continue_training": false, + "index_start" : 0, + "dir_of_start_model" : " ", + "weighted_loss": false, + "is_loss_soft_dice": false, + "data_is_provided": false, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +It's important to mention that the value of n_classes for enhancement should be 3, as the model's output is a 3-channel +image. + +#### Page extraction + +```yaml +{ + "backbone_type" : "nontransformer", + "task": "segmentation", + "n_classes" : 2, + "n_epochs" : 4, + "input_height" : 448, + "input_width" : 224, + "weight_decay" : 1e-6, + "n_batch" : 1, + "learning_rate": 1e-4, + "patches" : false, + "pretraining" : true, + "augmentation" : false, + "flip_aug" : false, + "blur_aug" : false, + "scaling" : true, + "degrading": false, + "brightening": false, + "binarization" : false, + "scaling_bluring" : false, + "scaling_binarization" : false, + "scaling_flip" : false, + "rotation": false, + "rotation_not_90": false, + "blur_k" : ["blur","guass","median"], + "scales" : [0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.4], + "brightness" : [1.3, 1.5, 1.7, 2], + "degrade_scales" : [0.2, 0.4], + "flip_index" : [0, 1, -1], + "thetha" : [10, -10], + "continue_training": false, + "index_start" : 0, + "dir_of_start_model" : " ", + "weighted_loss": false, + "is_loss_soft_dice": false, + "data_is_provided": false, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` + +For page segmentation (or printspace or border segmentation), the model needs to view the input image in its entirety, +hence the patches parameter should be set to false. + +#### layout segmentation +An example config json file for layout segmentation with 5 classes (including background) can be like this: + +```yaml +{ + "backbone_type" : "transformer", + "task": "segmentation", + "n_classes" : 5, + "n_epochs" : 4, + "input_height" : 448, + "input_width" : 224, + "weight_decay" : 1e-6, + "n_batch" : 1, + "learning_rate": 1e-4, + "patches" : true, + "pretraining" : true, + "augmentation" : true, + "flip_aug" : false, + "blur_aug" : false, + "scaling" : true, + "degrading": false, + "brightening": false, + "binarization" : false, + "scaling_bluring" : false, + "scaling_binarization" : false, + "scaling_flip" : false, + "rotation": false, + "rotation_not_90": false, + "transformer_num_patches_xy": [7, 14], + "transformer_patchsize_x": 1, + "transformer_patchsize_y": 1, + "transformer_projection_dim": 64, + "transformer_mlp_head_units": [128, 64], + "transformer_layers": 8, + "transformer_num_heads": 4, + "transformer_cnn_first": true, + "blur_k" : ["blur","guass","median"], + "scales" : [0.6, 0.7, 0.8, 0.9, 1.1, 1.2, 1.4], + "brightness" : [1.3, 1.5, 1.7, 2], + "degrade_scales" : [0.2, 0.4], + "flip_index" : [0, 1, -1], + "thetha" : [10, -10], + "continue_training": false, + "index_start" : 0, + "dir_of_start_model" : " ", + "weighted_loss": false, + "is_loss_soft_dice": false, + "data_is_provided": false, + "dir_train": "./train", + "dir_eval": "./eval", + "dir_output": "./output" +} +``` +## Inference with the trained model + +### classification +For conducting inference with a trained model, you simply need to execute the following command line, specifying the +directory of the model and the image on which to perform inference: + +`python inference.py -m "model dir" -i "image" ` + +This will straightforwardly return the class of the image. + +### machine based reading order +To infer the reading order using a reading order model, we need a page XML file containing layout information but +without the reading order. We simply need to provide the model directory, the XML file, and the output directory. +The new XML file with the added reading order will be written to the output directory with the same name. +We need to run: + +`python inference.py -m "model dir" -xml "page xml file" -o "output dir to write new xml with reading order" ` + +### Segmentation (Textline, Binarization, Page extraction and layout) and enhancement +For conducting inference with a trained model for segmentation and enhancement you need to run the following command +line: + +`python inference.py -m "model dir" -i "image" -p -s "output image" ` + +Note that in the case of page extraction the -p flag is not needed. + +For segmentation or binarization tasks, if a ground truth (GT) label is available, the IoU evaluation metric can be +calculated for the output. To do this, you need to provide the GT label using the argument -gt. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..da164de --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,92 @@ +# Usage documentation +The command-line interface can be called like this: + +```sh +eynollah \ + -i | -di \ + -o \ + -m \ + [OPTIONS] +``` + +## Processing options +The following options can be used to further configure the processing: + +| option | description | +|-------------------|:-------------------------------------------------------------------------------| +| `-fl` | full layout analysis including all steps and segmentation classes | +| `-light` | lighter and faster but simpler method for main region detection and deskewing | +| `-tab` | apply table detection | +| `-ae` | apply enhancement (the resulting image is saved to the output directory) | +| `-as` | apply scaling | +| `-cl` | apply contour detection for curved text lines instead of bounding boxes | +| `-ib` | apply binarization (the resulting image is saved to the output directory) | +| `-ep` | enable plotting (MUST always be used with `-sl`, `-sd`, `-sa`, `-si` or `-ae`) | +| `-eoi` | extract only images to output directory (other processing will not be done) | +| `-ho` | ignore headers for reading order dectection | +| `-si ` | save image regions detected to this directory | +| `-sd ` | save deskewed image to this directory | +| `-sl ` | save layout prediction as plot to this directory | +| `-sp ` | save cropped page image to this directory | +| `-sa ` | save all (plot, enhanced/binary image, layout) to this directory | + +If no option is set, the tool performs detection of main regions (background, text, images, separators and marginals). + +### `--full-layout` vs `--no-full-layout` + +Here are the difference in elements detected depending on the `--full-layout`/`--no-full-layout` command line flags: + +| | `--full-layout` | `--no-full-layout` | +|--------------------------|-----------------|--------------------| +| reading order | x | x | +| header regions | x | - | +| text regions | x | x | +| text regions / text line | x | x | +| drop-capitals | x | - | +| marginals | x | x | +| marginals / text line | x | x | +| image region | x | x | + +## Use as OCR-D processor +Eynollah ships with a CLI interface to be used as [OCR-D](https://ocr-d.de) processor that is described in +[`ocrd-tool.json`](https://github.com/qurator-spk/eynollah/tree/main/src/eynollah/ocrd-tool.json). + +The source image file group with (preferably) RGB images should be used as input for Eynollah like this: + +``` +ocrd-eynollah-segment -I OCR-D-IMG -O SEG-LINE -P models +``` + +Any image referenced by `@imageFilename` in PAGE-XML is passed on directly to Eynollah as a processor, so that e.g. + +``` +ocrd-eynollah-segment -I OCR-D-IMG-BIN -O SEG-LINE -P models +``` + +uses the original (RGB) image despite any binarization that may have occured in previous OCR-D processing steps. + +## Use with Docker +TODO + +## Hints +* The best output quality is produced when RGB images are used as input rather than greyscale or binarized images. +* If none of the parameters is set to `true`, the tool will perform a layout detection of main regions (background, +text, images, separators and marginals). An advantage of this tool is that it tries to extract main text regions +separately as much as possible. +* If you set `-ae` (**a**llow image **e**nhancement) parameter to `true`, the tool will first check the ppi +(pixel-per-inch) of the image and when it is less than 300, the tool will resize it and only then image enhancement will +occur. Image enhancement can also take place without this option, but by setting this option to `true`, the layout xml +data (e.g. coordinates) will be based on the resized and enhanced image instead of the original image. +* For some documents, while the quality is good, their scale is very large, and the performance of tool decreases. In +such cases you can set `-as` (**a**llow **s**caling) to `true`. With this option enabled, the tool will try to rescale +the image and only then the layout detection process will begin. +* If you care about drop capitals (initials) and headings, you can set `-fl` (**f**ull **l**ayout) to `true`. With this +setting, the tool can currently distinguish 7 document layout classes/elements. +* In cases where the document includes curved headers or curved lines, rectangular bounding boxes for textlines will not +be a great option. In such cases it is strongly recommended setting the flag `-cl` (**c**urved **l**ines) to `true` to +find contours of curved lines instead of rectangular bounding boxes. Be advised that enabling this option increases the +processing time of the tool. +* To crop and save image regions inside the document, set the parameter `-si` (**s**ave **i**mages) to true and provide +a directory path to store the extracted images. +* To extract only images from a document, set the parameter `-eoi` (**e**xtract **o**nly **i**mages). Choosing this +option disables any other processing. To save the cropped images add `-ep` and `-si`. diff --git a/requirements.txt b/requirements.txt index 2e1fc79..9ed0584 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,6 +3,5 @@ ocrd >= 3.3.0 numpy <1.24.0 scikit-learn >= 0.23.2 tensorflow < 2.13 -imutils >= 0.5.3 numba <= 0.58.1 loky diff --git a/src/eynollah/cli.py b/src/eynollah/cli.py index ecdfa3a..ef0f75e 100644 --- a/src/eynollah/cli.py +++ b/src/eynollah/cli.py @@ -324,6 +324,12 @@ def layout(image, out, overwrite, dir_in, model, save_images, save_layout, save_ help="directory of images", type=click.Path(exists=True, file_okay=False), ) +@click.option( + "--dir_in_bin", + "-dib", + help="directory of binarized images. This should be given if you want to do prediction based on both rgb and bin images. And all bin images are png files", + type=click.Path(exists=True, file_okay=False), +) @click.option( "--out", "-o", @@ -337,6 +343,12 @@ def layout(image, out, overwrite, dir_in, model, save_images, save_layout, save_ help="directory of xmls", type=click.Path(exists=True, file_okay=False), ) +@click.option( + "--dir_out_image_text", + "-doit", + help="directory of images with predicted text", + type=click.Path(exists=True, file_okay=False), +) @click.option( "--model", "-m", @@ -362,6 +374,18 @@ def layout(image, out, overwrite, dir_in, model, save_images, save_layout, save_ is_flag=True, help="if this parameter set to true, cropped textline images will not be masked with textline contour.", ) +@click.option( + "--draw_texts_on_image", + "-dtoi/-ndtoi", + is_flag=True, + help="if this parameter set to true, the predicted texts will be displayed on an image.", +) +@click.option( + "--prediction_with_both_of_rgb_and_bin", + "-brb/-nbrb", + is_flag=True, + help="If this parameter is set to True, the prediction will be performed using both RGB and binary images. However, this does not necessarily improve results; it may be beneficial for certain document images.", +) @click.option( "--log_level", "-l", @@ -369,18 +393,22 @@ def layout(image, out, overwrite, dir_in, model, save_images, save_layout, save_ help="Override log level globally to this", ) -def ocr(dir_in, out, dir_xmls, model, tr_ocr, export_textline_images_and_text, do_not_mask_with_textline_contour, log_level): +def ocr(dir_in, dir_in_bin, out, dir_xmls, dir_out_image_text, model, tr_ocr, export_textline_images_and_text, do_not_mask_with_textline_contour, draw_texts_on_image, prediction_with_both_of_rgb_and_bin, log_level): initLogging() if log_level: getLogger('eynollah').setLevel(getLevelName(log_level)) eynollah_ocr = Eynollah_ocr( dir_xmls=dir_xmls, + dir_out_image_text=dir_out_image_text, dir_in=dir_in, + dir_in_bin=dir_in_bin, dir_out=out, dir_models=model, tr_ocr=tr_ocr, export_textline_images_and_text=export_textline_images_and_text, do_not_mask_with_textline_contour=do_not_mask_with_textline_contour, + draw_texts_on_image=draw_texts_on_image, + prediction_with_both_of_rgb_and_bin=prediction_with_both_of_rgb_and_bin, ) eynollah_ocr.run() diff --git a/src/eynollah/eynollah.py b/src/eynollah/eynollah.py index ef8bcc6..9b8bb36 100644 --- a/src/eynollah/eynollah.py +++ b/src/eynollah/eynollah.py @@ -8,6 +8,7 @@ document layout analysis (segmentation) with output in PAGE-XML from logging import Logger from difflib import SequenceMatcher as sq +from PIL import Image, ImageDraw, ImageFont import math import os import sys @@ -383,7 +384,7 @@ class Eynollah: if uint8: key += '_uint8' return self._imgs[key].copy() - + def isNaN(self, num): return num != num @@ -534,7 +535,7 @@ class Eynollah: num_column_is_classified = True return img_new, num_column_is_classified - + def calculate_width_height_by_columns_1_2(self, img, num_col, width_early, label_p_pred): self.logger.debug("enter calculate_width_height_by_columns") if num_col == 1: @@ -687,7 +688,7 @@ class Eynollah: label_p_pred = self.model_classifier.predict(img_in, verbose=0) num_col = np.argmax(label_p_pred[0]) + 1 - + if num_col > self.num_col_upper: num_col = self.num_col_upper label_p_pred = [np.ones(6)] @@ -697,7 +698,7 @@ class Eynollah: else: num_col = self.num_col_upper label_p_pred = [np.ones(6)] - + self.logger.info("Found %d columns (%s)", num_col, np.around(label_p_pred, decimals=5)) if not self.extract_only_images: if dpi < DPI_THRESHOLD: @@ -798,13 +799,13 @@ class Eynollah: label_p_pred = model.predict(img[np.newaxis], verbose=0) seg = np.argmax(label_p_pred, axis=3)[0] - + if thresholding_for_artificial_class_in_light_version: seg_art = label_p_pred[0,:,:,2] - + seg_art[seg_art<0.2] = 0 seg_art[seg_art>0] =1 - + seg[seg_art==1]=2 seg_color = np.repeat(seg[:, :, np.newaxis], 3, axis=2) prediction_true = resize_image(seg_color, img_h_page, img_w_page).astype(np.uint8) @@ -986,31 +987,31 @@ class Eynollah: def do_padding_with_scale(self, img, scale): h_n = int(img.shape[0]*scale) w_n = int(img.shape[1]*scale) - + channel0_avg = int( np.mean(img[:,:,0]) ) channel1_avg = int( np.mean(img[:,:,1]) ) channel2_avg = int( np.mean(img[:,:,2]) ) - + h_diff = img.shape[0] - h_n w_diff = img.shape[1] - w_n - + h_start = int(0.5 * h_diff) w_start = int(0.5 * w_diff) - + img_res = resize_image(img, h_n, w_n) #label_res = resize_image(label, h_n, w_n) - + img_scaled_padded = np.copy(img) - + #label_scaled_padded = np.zeros(label.shape) - + img_scaled_padded[:,:,0] = channel0_avg img_scaled_padded[:,:,1] = channel1_avg img_scaled_padded[:,:,2] = channel2_avg - + img_scaled_padded[h_start:h_start+h_n, w_start:w_start+w_n,:] = img_res[:,:,:] #label_scaled_padded[h_start:h_start+h_n, w_start:w_start+w_n,:] = label_res[:,:,:] - + return img_scaled_padded#, label_scaled_padded def do_prediction_new_concept_scatter_nd( @@ -1031,18 +1032,18 @@ class Eynollah: label_p_pred = model.predict(img[np.newaxis], verbose=0) seg = np.argmax(label_p_pred, axis=3)[0] - + if thresholding_for_artificial_class_in_light_version: #seg_text = label_p_pred[0,:,:,1] #seg_text[seg_text<0.2] =0 #seg_text[seg_text>0] =1 #seg[seg_text==1]=1 - + seg_art = label_p_pred[0,:,:,4] seg_art[seg_art<0.2] =0 seg_art[seg_art>0] =1 seg[seg_art==1]=4 - + seg_color = np.repeat(seg[:, :, np.newaxis], 3, axis=2) prediction_true = resize_image(seg_color, img_h_page, img_w_page).astype(np.uint8) return prediction_true @@ -1060,10 +1061,10 @@ class Eynollah: img = img.astype(np.float16) img_h = img.shape[0] img_w = img.shape[1] - + stride_x = img_width_model - 100 stride_y = img_height_model - 100 - + one_tensor = tf.ones_like(img) img_patches, one_patches = tf.image.extract_patches( images=[img, one_tensor], @@ -1082,7 +1083,7 @@ class Eynollah: y = tf.range(img.shape[0]) x, y = tf.meshgrid(x, y) indices = tf.stack([y, x], axis=-1) - + indices_patches = tf.image.extract_patches( images=tf.expand_dims(indices, axis=0), sizes=[1, img_height_model, img_width_model, 1], @@ -1094,21 +1095,21 @@ class Eynollah: img_height_model, img_width_model, 2)) margin_y = int( 0.5 * (img_height_model - stride_y) ) margin_x = int( 0.5 * (img_width_model - stride_x) ) - + mask_margin = np.zeros((img_height_model, img_width_model)) mask_margin[margin_y:img_height_model - margin_y, margin_x:img_width_model - margin_x] = 1 - + indices_patches_array = indices_patches.numpy() for i in range(indices_patches_array.shape[0]): indices_patches_array[i,:,:,0] = indices_patches_array[i,:,:,0]*mask_margin indices_patches_array[i,:,:,1] = indices_patches_array[i,:,:,1]*mask_margin - + reconstructed = tf.scatter_nd( indices=indices_patches_array, updates=pred_patches, shape=(img.shape[0], img.shape[1], pred_patches.shape[-1])).numpy() - + prediction_true = np.argmax(reconstructed, axis=2).astype(np.uint8) gc.collect() return np.repeat(prediction_true[:, :, np.newaxis], 3, axis=2) @@ -1131,21 +1132,21 @@ class Eynollah: label_p_pred = model.predict(img[np.newaxis], verbose=0) seg = np.argmax(label_p_pred, axis=3)[0] - + if thresholding_for_artificial_class_in_light_version: #seg_text = label_p_pred[0,:,:,1] #seg_text[seg_text<0.2] =0 #seg_text[seg_text>0] =1 #seg[seg_text==1]=1 - + seg_art = label_p_pred[0,:,:,4] seg_art[seg_art<0.2] =0 seg_art[seg_art>0] =1 seg[seg_art==1]=4 - + seg_color = np.repeat(seg[:, :, np.newaxis], 3, axis=2) prediction_true = resize_image(seg_color, img_h_page, img_w_page).astype(np.uint8) - return prediction_true + return prediction_true , resize_image(label_p_pred[0, :, :, 1] , img_h_page, img_w_page) if img.shape[0] < img_height_model: img = resize_image(img, img_height_model, img.shape[1]) @@ -1161,6 +1162,7 @@ class Eynollah: img_h = img.shape[0] img_w = img.shape[1] prediction_true = np.zeros((img_h, img_w, 3)) + confidence_matrix = np.zeros((img_h, img_w)) mask_true = np.zeros((img_h, img_w)) nxf = img_w / float(width_mid) nyf = img_h / float(height_mid) @@ -1249,54 +1251,99 @@ class Eynollah: seg_in[0:-margin or None, 0:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + 0:index_y_u_in - margin, + index_x_d_in + 0:index_x_u_in - margin] = \ + label_p_pred[0, 0:-margin or None, + 0:-margin or None, + 1] elif i_batch == nxf - 1 and j_batch == nyf - 1: prediction_true[index_y_d_in + margin:index_y_u_in - 0, index_x_d_in + margin:index_x_u_in - 0] = \ seg_in[margin:, margin:, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - 0, + index_x_d_in + margin:index_x_u_in - 0] = \ + label_p_pred[0, margin:, + margin:, + 1] elif i_batch == 0 and j_batch == nyf - 1: prediction_true[index_y_d_in + margin:index_y_u_in - 0, index_x_d_in + 0:index_x_u_in - margin] = \ seg_in[margin:, 0:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - 0, + index_x_d_in + 0:index_x_u_in - margin] = \ + label_p_pred[0, margin:, + 0:-margin or None, + 1] elif i_batch == nxf - 1 and j_batch == 0: prediction_true[index_y_d_in + 0:index_y_u_in - margin, index_x_d_in + margin:index_x_u_in - 0] = \ seg_in[0:-margin or None, margin:, np.newaxis] + confidence_matrix[index_y_d_in + 0:index_y_u_in - margin, + index_x_d_in + margin:index_x_u_in - 0] = \ + label_p_pred[0, 0:-margin or None, + margin:, + 1] elif i_batch == 0 and j_batch != 0 and j_batch != nyf - 1: prediction_true[index_y_d_in + margin:index_y_u_in - margin, index_x_d_in + 0:index_x_u_in - margin] = \ seg_in[margin:-margin or None, 0:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - margin, + index_x_d_in + 0:index_x_u_in - margin] = \ + label_p_pred[0, margin:-margin or None, + 0:-margin or None, + 1] elif i_batch == nxf - 1 and j_batch != 0 and j_batch != nyf - 1: prediction_true[index_y_d_in + margin:index_y_u_in - margin, index_x_d_in + margin:index_x_u_in - 0] = \ seg_in[margin:-margin or None, margin:, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - margin, + index_x_d_in + margin:index_x_u_in - 0] = \ + label_p_pred[0, margin:-margin or None, + margin:, + 1] elif i_batch != 0 and i_batch != nxf - 1 and j_batch == 0: prediction_true[index_y_d_in + 0:index_y_u_in - margin, index_x_d_in + margin:index_x_u_in - margin] = \ seg_in[0:-margin or None, margin:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + 0:index_y_u_in - margin, + index_x_d_in + margin:index_x_u_in - margin] = \ + label_p_pred[0, 0:-margin or None, + margin:-margin or None, + 1] elif i_batch != 0 and i_batch != nxf - 1 and j_batch == nyf - 1: prediction_true[index_y_d_in + margin:index_y_u_in - 0, index_x_d_in + margin:index_x_u_in - margin] = \ seg_in[margin:, margin:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - 0, + index_x_d_in + margin:index_x_u_in - margin] = \ + label_p_pred[0, margin:, + margin:-margin or None, + 1] else: prediction_true[index_y_d_in + margin:index_y_u_in - margin, index_x_d_in + margin:index_x_u_in - margin] = \ seg_in[margin:-margin or None, margin:-margin or None, np.newaxis] + confidence_matrix[index_y_d_in + margin:index_y_u_in - margin, + index_x_d_in + margin:index_x_u_in - margin] = \ + label_p_pred[0, margin:-margin or None, + margin:-margin or None, + 1] indexer_inside_batch += 1 list_i_s = [] @@ -1311,7 +1358,7 @@ class Eynollah: prediction_true = prediction_true.astype(np.uint8) gc.collect() - return prediction_true + return prediction_true, confidence_matrix def extract_page(self): self.logger.debug("enter extract_page") @@ -1323,7 +1370,7 @@ class Eynollah: _, thresh = cv2.threshold(imgray, 0, 255, 0) thresh = cv2.dilate(thresh, KERNEL, iterations=3) contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) - + if len(contours)>0: cnt_size = np.array([cv2.contourArea(contours[j]) for j in range(len(contours))]) @@ -1379,7 +1426,7 @@ class Eynollah: else: box = [0, 0, img.shape[1], img.shape[0]] cropped_page, page_coord = crop_image_inside_box(box, img) - + self.logger.debug("exit early_page_for_num_of_column_classification") else: img = self.imread() @@ -1470,35 +1517,35 @@ class Eynollah: prediction_regions = resize_image(prediction_regions, img_height_h, img_width_h) self.logger.debug("exit extract_text_regions") return prediction_regions, prediction_regions2 - + def get_slopes_and_deskew_new_light2(self, contours, contours_par, textline_mask_tot, image_page_rotated, boxes, slope_deskew): - + polygons_of_textlines = return_contours_of_interested_region(textline_mask_tot,1,0.00001) M_main_tot = [cv2.moments(polygons_of_textlines[j]) for j in range(len(polygons_of_textlines))] cx_main_tot = [(M_main_tot[j]["m10"] / (M_main_tot[j]["m00"] + 1e-32)) for j in range(len(M_main_tot))] cy_main_tot = [(M_main_tot[j]["m01"] / (M_main_tot[j]["m00"] + 1e-32)) for j in range(len(M_main_tot))] - + args_textlines = np.array(range(len(polygons_of_textlines))) all_found_textline_polygons = [] slopes = [] all_box_coord =[] - + for index, con_region_ind in enumerate(contours_par): results = [cv2.pointPolygonTest(con_region_ind, (cx_main_tot[ind], cy_main_tot[ind]), False) for ind in args_textlines ] results = np.array(results) indexes_in = args_textlines[results==1] textlines_ins = [polygons_of_textlines[ind] for ind in indexes_in] - - all_found_textline_polygons.append(textlines_ins) + + all_found_textline_polygons.append(textlines_ins[::-1]) slopes.append(slope_deskew) - + _, crop_coor = crop_image_inside_box(boxes[index],image_page_rotated) all_box_coord.append(crop_coor) - + return all_found_textline_polygons, boxes, contours, contours_par, all_box_coord, np.array(range(len(contours_par))), slopes - + def get_slopes_and_deskew_new_light(self, contours, contours_par, textline_mask_tot, image_page_rotated, boxes, slope_deskew): if not len(contours): return [], [], [], [], [], [], [] @@ -1558,7 +1605,7 @@ class Eynollah: img_h = img_org.shape[0] img_w = img_org.shape[1] img = resize_image(img_org, int(img_org.shape[0] * scaler_h), int(img_org.shape[1] * scaler_w)) - + prediction_textline = self.do_prediction( use_patches, img, self.model_textline, marginal_of_patch_percent=0.15, n_batch_inference=3, @@ -1570,25 +1617,25 @@ class Eynollah: prediction_textline = resize_image(prediction_textline, img_h, img_w) textline_mask_tot_ea_art = (prediction_textline[:,:]==2)*1 - + old_art = np.copy(textline_mask_tot_ea_art) if not self.textline_light: textline_mask_tot_ea_art = textline_mask_tot_ea_art.astype('uint8') #textline_mask_tot_ea_art = cv2.dilate(textline_mask_tot_ea_art, KERNEL, iterations=1) prediction_textline[:,:][textline_mask_tot_ea_art[:,:]==1]=2 - + textline_mask_tot_ea_lines = (prediction_textline[:,:]==1)*1 textline_mask_tot_ea_lines = textline_mask_tot_ea_lines.astype('uint8') if not self.textline_light: textline_mask_tot_ea_lines = cv2.dilate(textline_mask_tot_ea_lines, KERNEL, iterations=1) - + prediction_textline[:,:][textline_mask_tot_ea_lines[:,:]==1]=1 if not self.textline_light: prediction_textline[:,:][old_art[:,:]==1]=2 - + prediction_textline_longshot = self.do_prediction(False, img, self.model_textline) prediction_textline_longshot_true_size = resize_image(prediction_textline_longshot, img_h, img_w) - + self.logger.debug('exit textline_contours') return ((prediction_textline[:, :, 0]==1).astype(np.uint8), (prediction_textline_longshot_true_size[:, :, 0]==1).astype(np.uint8)) @@ -1653,7 +1700,7 @@ class Eynollah: img_h_new = int(img.shape[0] / float(img.shape[1]) * img_w_new) img_resized = resize_image(img,img_h_new, img_w_new ) - prediction_regions_org = self.do_prediction_new_concept(True, img_resized, self.model_region) + prediction_regions_org, _ = self.do_prediction_new_concept(True, img_resized, self.model_region) prediction_regions_org = resize_image(prediction_regions_org,img_height_h, img_width_h ) image_page, page_coord, cont_page = self.extract_page() @@ -1696,15 +1743,15 @@ class Eynollah: """ test_poly_image = np.zeros((text_regions_p_true.shape[0], text_regions_p_true.shape[1])) test_poly_image = cv2.fillPoly(test_poly_image, pts=[ploy_img_ind], color=(1,1,1)) - + test_poly_image = test_poly_image + image_boundary_of_doc test_poly_image_intersected_area = ( test_poly_image[:,:]==2 )*1 - + test_poly_image_intersected_area = test_poly_image_intersected_area.sum() - + if test_poly_image_intersected_area==0: ##polygons_of_images_fin.append(ploy_img_ind) - + box = cv2.boundingRect(ploy_img_ind) _, page_coord_img = crop_image_inside_box(box, text_regions_p_true) # cont_page.append(np.array([[page_coord[2], page_coord[0]], @@ -1742,7 +1789,7 @@ class Eynollah: img_width_h = img_org.shape[1] #print(num_col_classifier,'num_col_classifier') - + if num_col_classifier == 1: img_w_new = 1000 elif num_col_classifier == 2: @@ -1757,21 +1804,21 @@ class Eynollah: img_w_new = 4000 img_h_new = img_w_new * img_org.shape[0] // img_org.shape[1] img_resized = resize_image(img,img_h_new, img_w_new ) - + t_bin = time.time() #if (not self.input_binary) or self.full_layout: #if self.input_binary: #img_bin = np.copy(img_resized) ###if (not self.input_binary and self.full_layout) or (not self.input_binary and num_col_classifier >= 30): ###prediction_bin = self.do_prediction(True, img_resized, self.model_bin, n_batch_inference=5) - + ####print("inside bin ", time.time()-t_bin) ###prediction_bin=prediction_bin[:,:,0] ###prediction_bin = (prediction_bin[:,:]==0)*1 ###prediction_bin = prediction_bin*255 - + ###prediction_bin =np.repeat(prediction_bin[:, :, np.newaxis], 3, axis=2) - + ###prediction_bin = prediction_bin.astype(np.uint16) ####img= np.copy(prediction_bin) ###img_bin = np.copy(prediction_bin) @@ -1787,15 +1834,15 @@ class Eynollah: else: img_bin = np.copy(img_resized) #print("inside 1 ", time.time()-t_in) - + ###textline_mask_tot_ea = self.run_textline(img_bin) self.logger.debug("detecting textlines on %s with %d colors", str(img_resized.shape), len(np.unique(img_resized))) textline_mask_tot_ea = self.run_textline(img_resized, num_col_classifier) textline_mask_tot_ea = resize_image(textline_mask_tot_ea,img_height_h, img_width_h ) - + #print(self.image_org.shape) #cv2.imwrite('out_13.png', self.image_page_org_size) - + #plt.imshwo(self.image_page_org_size) #plt.show() if not skip_layout_and_reading_order: @@ -1804,46 +1851,49 @@ class Eynollah: if self.image_org.shape[0]/self.image_org.shape[1] > 2.5: self.logger.debug("resized to %dx%d for %d cols", img_resized.shape[1], img_resized.shape[0], num_col_classifier) - prediction_regions_org = self.do_prediction_new_concept( + prediction_regions_org, confidence_matrix = self.do_prediction_new_concept( True, img_resized, self.model_region_1_2, n_batch_inference=1, thresholding_for_some_classes_in_light_version=True) else: prediction_regions_org = np.zeros((self.image_org.shape[0], self.image_org.shape[1], 3)) - prediction_regions_page = self.do_prediction_new_concept( + confidence_matrix = np.zeros((self.image_org.shape[0], self.image_org.shape[1])) + prediction_regions_page, confidence_matrix_page = self.do_prediction_new_concept( False, self.image_page_org_size, self.model_region_1_2, n_batch_inference=1, thresholding_for_artificial_class_in_light_version=True) ys = slice(*self.page_coord[0:2]) xs = slice(*self.page_coord[2:4]) prediction_regions_org[ys, xs] = prediction_regions_page + confidence_matrix[ys, xs] = confidence_matrix_page else: new_h = (900+ (num_col_classifier-3)*100) img_resized = resize_image(img_bin, int(new_h * img_bin.shape[0] /img_bin.shape[1]), new_h) self.logger.debug("resized to %dx%d (new_h=%d) for %d cols", img_resized.shape[1], img_resized.shape[0], new_h, num_col_classifier) - prediction_regions_org = self.do_prediction_new_concept( + prediction_regions_org, confidence_matrix = self.do_prediction_new_concept( True, img_resized, self.model_region_1_2, n_batch_inference=2, thresholding_for_some_classes_in_light_version=True) ###prediction_regions_org = self.do_prediction(True, img_bin, self.model_region, n_batch_inference=3, thresholding_for_some_classes_in_light_version=True) #print("inside 3 ", time.time()-t_in) #plt.imshow(prediction_regions_org[:,:,0]) #plt.show() - - prediction_regions_org = resize_image(prediction_regions_org,img_height_h, img_width_h ) - img_bin = resize_image(img_bin,img_height_h, img_width_h ) + + prediction_regions_org = resize_image(prediction_regions_org, img_height_h, img_width_h ) + confidence_matrix = resize_image(confidence_matrix, img_height_h, img_width_h ) + img_bin = resize_image(img_bin, img_height_h, img_width_h ) prediction_regions_org=prediction_regions_org[:,:,0] - + mask_lines_only = (prediction_regions_org[:,:] ==3)*1 mask_texts_only = (prediction_regions_org[:,:] ==1)*1 mask_texts_only = mask_texts_only.astype('uint8') - + ##if num_col_classifier == 1 or num_col_classifier == 2: ###mask_texts_only = cv2.erode(mask_texts_only, KERNEL, iterations=1) ##mask_texts_only = cv2.dilate(mask_texts_only, KERNEL, iterations=1) - + mask_texts_only = cv2.dilate(mask_texts_only, kernel=np.ones((2,2), np.uint8), iterations=1) mask_images_only=(prediction_regions_org[:,:] ==2)*1 - + polygons_lines_xml, hir_lines_xml = return_contours_of_image(mask_lines_only) test_khat = np.zeros(prediction_regions_org.shape) test_khat = cv2.fillPoly(test_khat, pts=polygons_lines_xml, color=(1,1,1)) @@ -1859,38 +1909,38 @@ class Eynollah: polygons_lines_xml = filter_contours_area_of_image( mask_lines_only, polygons_lines_xml, hir_lines_xml, max_area=1, min_area=0.00001) - + test_khat = np.zeros(prediction_regions_org.shape) test_khat = cv2.fillPoly(test_khat, pts = polygons_lines_xml, color=(1,1,1)) - + #plt.imshow(test_khat[:,:]) #plt.show() #sys.exit() - + polygons_of_only_texts = return_contours_of_interested_region(mask_texts_only,1,0.00001) ##polygons_of_only_texts = self.dilate_textregions_contours(polygons_of_only_texts) polygons_of_only_lines = return_contours_of_interested_region(mask_lines_only,1,0.00001) - + text_regions_p_true = np.zeros(prediction_regions_org.shape) text_regions_p_true = cv2.fillPoly(text_regions_p_true, pts=polygons_of_only_lines, color=(3,3,3)) - + text_regions_p_true[:,:][mask_images_only[:,:] == 1] = 2 text_regions_p_true = cv2.fillPoly(text_regions_p_true, pts = polygons_of_only_texts, color=(1,1,1)) - + #plt.imshow(textline_mask_tot_ea) #plt.show() - + textline_mask_tot_ea[(text_regions_p_true==0) | (text_regions_p_true==4) ] = 0 - + #plt.imshow(textline_mask_tot_ea) #plt.show() #print("inside 4 ", time.time()-t_in) self.logger.debug("exit get_regions_light_v") - return text_regions_p_true, erosion_hurts, polygons_lines_xml, textline_mask_tot_ea, img_bin + return text_regions_p_true, erosion_hurts, polygons_lines_xml, textline_mask_tot_ea, img_bin, confidence_matrix else: img_bin = resize_image(img_bin,img_height_h, img_width_h ) self.logger.debug("exit get_regions_light_v") - return None, erosion_hurts, None, textline_mask_tot_ea, img_bin + return None, erosion_hurts, None, textline_mask_tot_ea, img_bin, None def get_regions_from_xy_2models(self,img,is_image_enhanced, num_col_classifier): self.logger.debug("enter get_regions_from_xy_2models") @@ -1910,14 +1960,14 @@ class Eynollah: #plt.show() prediction_regions_org_y = prediction_regions_org_y[:,:,0] mask_zeros_y = (prediction_regions_org_y[:,:]==0)*1 - + ##img_only_regions_with_sep = ( (prediction_regions_org_y[:,:] != 3) & (prediction_regions_org_y[:,:] != 0) )*1 img_only_regions_with_sep = (prediction_regions_org_y == 1).astype(np.uint8) try: img_only_regions = cv2.erode(img_only_regions_with_sep[:,:], KERNEL, iterations=20) _, _ = find_num_col(img_only_regions, num_col_classifier, self.tables, multiplier=6.0) img = resize_image(img_org, int(img_org.shape[0]), int(img_org.shape[1]*(1.2 if is_image_enhanced else 1))) - + prediction_regions_org = self.do_prediction(True, img, self.model_region) prediction_regions_org = resize_image(prediction_regions_org, img_height_h, img_width_h ) @@ -1959,16 +2009,16 @@ class Eynollah: ratio_x=1 img = resize_image(prediction_bin, int(img_org.shape[0]*ratio_y), int(img_org.shape[1]*ratio_x)) - + prediction_regions_org = self.do_prediction(True, img, self.model_region) prediction_regions_org = resize_image(prediction_regions_org, img_height_h, img_width_h ) prediction_regions_org=prediction_regions_org[:,:,0] - + mask_lines_only=(prediction_regions_org[:,:]==3)*1 - + mask_texts_only=(prediction_regions_org[:,:]==1)*1 mask_images_only=(prediction_regions_org[:,:]==2)*1 - + polygons_lines_xml, hir_lines_xml = return_contours_of_image(mask_lines_only) polygons_lines_xml = filter_contours_area_of_image( mask_lines_only, polygons_lines_xml, hir_lines_xml, max_area=1, min_area=0.00001) @@ -2001,36 +2051,36 @@ class Eynollah: prediction_regions_org = self.do_prediction(True, img, self.model_region) prediction_regions_org = resize_image(prediction_regions_org, img_height_h, img_width_h ) prediction_regions_org=prediction_regions_org[:,:,0] - + #mask_lines_only=(prediction_regions_org[:,:]==3)*1 #img = resize_image(img_org, int(img_org.shape[0]*1), int(img_org.shape[1]*1)) - + #prediction_regions_org = self.do_prediction(True, img, self.model_region) - + #prediction_regions_org = resize_image(prediction_regions_org, img_height_h, img_width_h ) - + #prediction_regions_org = prediction_regions_org[:,:,0] - + #prediction_regions_org[(prediction_regions_org[:,:] == 1) & (mask_zeros_y[:,:] == 1)]=0 - - + + mask_lines_only = (prediction_regions_org == 3)*1 mask_texts_only = (prediction_regions_org == 1)*1 mask_images_only= (prediction_regions_org == 2)*1 - + polygons_lines_xml, hir_lines_xml = return_contours_of_image(mask_lines_only) polygons_lines_xml = filter_contours_area_of_image( mask_lines_only, polygons_lines_xml, hir_lines_xml, max_area=1, min_area=0.00001) - + polygons_of_only_texts = return_contours_of_interested_region(mask_texts_only,1,0.00001) polygons_of_only_lines = return_contours_of_interested_region(mask_lines_only,1,0.00001) - + text_regions_p_true = np.zeros(prediction_regions_org.shape) text_regions_p_true = cv2.fillPoly(text_regions_p_true, pts = polygons_of_only_lines, color=(3,3,3)) - + text_regions_p_true[:,:][mask_images_only[:,:] == 1] = 2 text_regions_p_true = cv2.fillPoly(text_regions_p_true, pts = polygons_of_only_texts, color=(1,1,1)) - + erosion_hurts = True self.logger.debug("exit get_regions_from_xy_2models") return text_regions_p_true, erosion_hurts, polygons_lines_xml @@ -2140,7 +2190,7 @@ class Eynollah: order_text_new = [] for iii in range(len(order_of_texts_tot)): order_text_new.append(np.where(np.array(order_of_texts_tot) == iii)[0][0]) - + except Exception as why: self.logger.error(why) arg_text_con = [] @@ -2155,7 +2205,7 @@ class Eynollah: arg_text_con.append(jj) check_if_textregion_located_in_a_box = True break - + if not check_if_textregion_located_in_a_box: dists_tr_from_box = [math.sqrt((cx_text_only[ii] - boxes[jj][1]) ** 2 + (cy_text_only[ii] - boxes[jj][2]) ** 2) @@ -2242,7 +2292,7 @@ class Eynollah: order_text_new = [] for iii in range(len(order_of_texts_tot)): order_text_new.append(np.where(np.array(order_of_texts_tot) == iii)[0][0]) - + self.logger.debug("exit do_order_of_regions_full_layout") return order_text_new, id_of_texts_tot @@ -2314,7 +2364,7 @@ class Eynollah: order_text_new = [] for iii in range(len(order_of_texts_tot)): order_text_new.append(np.where(np.array(order_of_texts_tot) == iii)[0][0]) - + except Exception as why: self.logger.error(why) arg_text_con = [] @@ -2371,14 +2421,14 @@ class Eynollah: ref_point += len(id_of_texts) order_of_texts_tot = [] - + for tj1 in range(len(contours_only_text_parent)): order_of_texts_tot.append(int(order_by_con_main[tj1])) order_text_new = [] for iii in range(len(order_of_texts_tot)): order_text_new.append(np.where(np.array(order_of_texts_tot) == iii)[0][0]) - + self.logger.debug("exit do_order_of_regions_no_full_layout") return order_text_new, id_of_texts_tot @@ -2397,7 +2447,7 @@ class Eynollah: contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) cnt_size = np.array([cv2.contourArea(contours[j]) for j in range(len(contours))]) - + contours_new = [] for i in range(len(contours)): x, y, w, h = cv2.boundingRect(contours[i]) @@ -2413,16 +2463,16 @@ class Eynollah: peaks, _ = find_peaks(layout_contour_sum_diff_smoothed, height=0) peaks= peaks[layout_contour_sum_diff_smoothed[peaks]>4] - + for j in range(len(peaks)): layout_contour[:,peaks[j]-3+1:peaks[j]+1+3] = 0 - + layout_contour=cv2.erode(layout_contour[:,:], KERNEL, iterations=5) layout_contour=cv2.dilate(layout_contour[:,:], KERNEL, iterations=5) - + layout_contour =np.repeat(layout_contour[:, :, np.newaxis], 3, axis=2) layout_contour = layout_contour.astype(np.uint8) - + imgray = cv2.cvtColor(layout_contour, cv2.COLOR_BGR2GRAY ) _, thresh = cv2.threshold(imgray, 0, 255, 0) @@ -2436,7 +2486,7 @@ class Eynollah: table_pixels_masked_from_early_pre = only_recent_contour_image * table_prediction_early iou_in = 100. * table_pixels_masked_from_early_pre.sum() / only_recent_contour_image.sum() #print(iou_in,'iou_in_in1') - + if iou_in>30: layout_org= cv2.fillPoly(layout_org, pts=[contours_sep[ji]], color=3 * (pixel_table,)) else: @@ -2448,7 +2498,7 @@ class Eynollah: if num_col_classifier>=2: only_recent_contour_image = np.zeros((layout.shape[0],layout.shape[1])) only_recent_contour_image= cv2.fillPoly(only_recent_contour_image,pts=[contours[i]] ,color=(1,1,1)) - + table_pixels_masked_from_early_pre = only_recent_contour_image * table_prediction_early iou_in = 100. * table_pixels_masked_from_early_pre.sum() / only_recent_contour_image.sum() #print(iou_in,'iou_in') @@ -2458,7 +2508,7 @@ class Eynollah: pass else: layout_org= cv2.fillPoly(layout_org, pts=[contours[i]], color=3 * (pixel_table,)) - + return layout_org, contours_new def delete_separator_around(self, spliter_y,peaks_neg,image_by_region, pixel_line, pixel_table): @@ -2474,7 +2524,7 @@ class Eynollah: image_by_region[ys,xs,0][image_by_region[ys,xs,0]==pixel_line] = 0 image_by_region[ys,xs,0][image_by_region[ys,xs,1]==pixel_line] = 0 image_by_region[ys,xs,0][image_by_region[ys,xs,2]==pixel_line] = 0 - + image_by_region[ys,xs,0][image_by_region[ys,xs,0]==pixel_table] = 0 image_by_region[ys,xs,0][image_by_region[ys,xs,1]==pixel_table] = 0 image_by_region[ys,xs,0][image_by_region[ys,xs,2]==pixel_table] = 0 @@ -2496,14 +2546,14 @@ class Eynollah: pixel_table =10 image_revised_1 = self.delete_separator_around(spliter_y, peaks_neg_tot, image_revised, pixel_line, pixel_table) - + try: image_revised_1[:,:30][image_revised_1[:,:30]==pixel_line] = 0 image_revised_1[:,-30:][image_revised_1[:,-30:]==pixel_line] = 0 except: pass boxes = np.array(boxes, dtype=int) # to be on the safe side - + img_comm_e = np.zeros(image_revised_1.shape) img_comm = np.repeat(img_comm_e[:, :, np.newaxis], 3, axis=2) @@ -2523,7 +2573,7 @@ class Eynollah: img_comm = cv2.fillPoly(img_comm, pts = main_contours, color = (indiv, indiv, indiv)) img_comm = img_comm.astype(np.uint8) - + if not self.isNaN(slope_mean_hor): image_revised_last = np.zeros((image_regions_eraly_p.shape[0], image_regions_eraly_p.shape[1],3)) for i in range(len(boxes)): @@ -2599,13 +2649,13 @@ class Eynollah: box_xs = slice(*boxes[i][0:2]) image_box = img_comm[box_ys, box_xs] image_revised_last[box_ys, box_xs] = image_box - + if num_col_classifier==1: img_tables_col_1 = (image_revised_last[:,:,0] == pixel_table).astype(np.uint8) contours_table_col1, _ = return_contours_of_image(img_tables_col_1) - + _,_ ,_ , _, y_min_tab_col1 ,y_max_tab_col1, _= find_new_features_of_contours(contours_table_col1) - + if len(y_min_tab_col1)>0: for ijv in range(len(y_min_tab_col1)): image_revised_last[int(y_min_tab_col1[ijv]):int(y_max_tab_col1[ijv]),:,:]=pixel_table @@ -2615,14 +2665,14 @@ class Eynollah: if self.full_layout: return self.do_order_of_regions_full_layout(*args, **kwargs) return self.do_order_of_regions_no_full_layout(*args, **kwargs) - + def get_tables_from_model(self, img, num_col_classifier): img_org = np.copy(img) img_height_h = img_org.shape[0] img_width_h = img_org.shape[1] patches = False if self.light_version: - prediction_table = self.do_prediction_new_concept(patches, img, self.model_table) + prediction_table, _ = self.do_prediction_new_concept(patches, img, self.model_table) prediction_table = prediction_table.astype(np.int16) return prediction_table[:,:,0] else: @@ -2630,16 +2680,16 @@ class Eynollah: prediction_table = self.do_prediction(patches, img, self.model_table) pre_updown = self.do_prediction(patches, cv2.flip(img[:,:,:], -1), self.model_table) pre_updown = cv2.flip(pre_updown, -1) - + prediction_table[:,:,0][pre_updown[:,:,0]==1]=1 prediction_table = prediction_table.astype(np.int16) - + elif num_col_classifier ==2: height_ext = 0 # img.shape[0] // 4 h_start = height_ext // 2 width_ext = img.shape[1] // 8 w_start = width_ext // 2 - + img_new = np.zeros((img.shape[0] + height_ext, img.shape[1] + width_ext, img.shape[2])).astype(float) @@ -2650,10 +2700,10 @@ class Eynollah: prediction_ext = self.do_prediction(patches, img_new, self.model_table) pre_updown = self.do_prediction(patches, cv2.flip(img_new[:,:,:], -1), self.model_table) pre_updown = cv2.flip(pre_updown, -1) - + prediction_table = prediction_ext[ys, xs] prediction_table_updown = pre_updown[ys, xs] - + prediction_table[:,:,0][prediction_table_updown[:,:,0]==1]=1 prediction_table = prediction_table.astype(np.int16) elif num_col_classifier ==1: @@ -2661,7 +2711,7 @@ class Eynollah: h_start = height_ext // 2 width_ext = img.shape[1] // 4 w_start = width_ext // 2 - + img_new =np.zeros((img.shape[0] + height_ext, img.shape[1] + width_ext, img.shape[2])).astype(float) @@ -2672,10 +2722,10 @@ class Eynollah: prediction_ext = self.do_prediction(patches, img_new, self.model_table) pre_updown = self.do_prediction(patches, cv2.flip(img_new[:,:,:], -1), self.model_table) pre_updown = cv2.flip(pre_updown, -1) - + prediction_table = prediction_ext[ys, xs] prediction_table_updown = pre_updown[ys, xs] - + prediction_table[:,:,0][prediction_table_updown[:,:,0]==1]=1 prediction_table = prediction_table.astype(np.int16) else: @@ -2687,23 +2737,23 @@ class Eynollah: pre_full = self.do_prediction(patches, img[:,:,:], self.model_table) pre_updown = self.do_prediction(patches, cv2.flip(img[:,:,:], -1), self.model_table) pre_updown = cv2.flip(pre_updown, -1) - + prediction_table_full_erode = cv2.erode(pre_full[:,:,0], KERNEL, iterations=4) prediction_table_full_erode = cv2.dilate(prediction_table_full_erode, KERNEL, iterations=4) - + prediction_table_full_updown_erode = cv2.erode(pre_updown[:,:,0], KERNEL, iterations=4) prediction_table_full_updown_erode = cv2.dilate(prediction_table_full_updown_erode, KERNEL, iterations=4) prediction_table[:,0:img_w_half,:] = pre1[:,:,:] prediction_table[:,img_w_half:,:] = pre2[:,:,:] - + prediction_table[:,:,0][prediction_table_full_erode[:,:]==1]=1 prediction_table[:,:,0][prediction_table_full_updown_erode[:,:]==1]=1 prediction_table = prediction_table.astype(np.int16) - + #prediction_table_erode = cv2.erode(prediction_table[:,:,0], self.kernel, iterations=6) #prediction_table_erode = cv2.dilate(prediction_table_erode, self.kernel, iterations=6) - + prediction_table_erode = cv2.erode(prediction_table[:,:,0], KERNEL, iterations=20) prediction_table_erode = cv2.dilate(prediction_table_erode, KERNEL, iterations=20) return prediction_table_erode.astype(np.int16) @@ -2729,14 +2779,14 @@ class Eynollah: table_prediction = self.get_tables_from_model(image_page, num_col_classifier) else: table_prediction = np.zeros((image_page.shape[0], image_page.shape[1])).astype(np.int16) - + if self.plotter: self.plotter.save_page_image(image_page) text_regions_p_1 = text_regions_p_1[page_coord[0] : page_coord[1], page_coord[2] : page_coord[3]] textline_mask_tot_ea = textline_mask_tot_ea[page_coord[0] : page_coord[1], page_coord[2] : page_coord[3]] img_bin_light = img_bin_light[page_coord[0] : page_coord[1], page_coord[2] : page_coord[3]] - + mask_images = (text_regions_p_1[:, :] == 2) * 1 mask_images = mask_images.astype(np.uint8) mask_images = cv2.erode(mask_images[:, :], KERNEL, iterations=10) @@ -2744,13 +2794,13 @@ class Eynollah: mask_lines = mask_lines.astype(np.uint8) img_only_regions_with_sep = ((text_regions_p_1[:, :] != 3) & (text_regions_p_1[:, :] != 0)) * 1 img_only_regions_with_sep = img_only_regions_with_sep.astype(np.uint8) - + #print("inside graphics 2 ", time.time() - t_in_gr) if erosion_hurts: img_only_regions = np.copy(img_only_regions_with_sep[:,:]) else: img_only_regions = cv2.erode(img_only_regions_with_sep[:,:], KERNEL, iterations=6) - + ##print(img_only_regions.shape,'img_only_regions') ##plt.imshow(img_only_regions[:,:]) ##plt.show() @@ -2766,7 +2816,7 @@ class Eynollah: #print("inside graphics 3 ", time.time() - t_in_gr) return (num_col, num_col_classifier, img_only_regions, page_coord, image_page, mask_images, mask_lines, text_regions_p_1, cont_page, table_prediction, textline_mask_tot_ea, img_bin_light) - + def run_graphics_and_columns_without_layout(self, textline_mask_tot_ea, img_bin_light): #print(text_regions_p_1.shape, 'text_regions_p_1 shape run graphics') #print(erosion_hurts, 'erosion_hurts') @@ -2781,10 +2831,10 @@ class Eynollah: image_page, page_coord, cont_page = self.extract_page() #print("inside graphics 1 ", time.time() - t_in_gr) - + textline_mask_tot_ea = textline_mask_tot_ea[page_coord[0] : page_coord[1], page_coord[2] : page_coord[3]] img_bin_light = img_bin_light[page_coord[0] : page_coord[1], page_coord[2] : page_coord[3]] - + return page_coord, image_page, textline_mask_tot_ea, img_bin_light, cont_page def run_graphics_and_columns( @@ -2801,12 +2851,12 @@ class Eynollah: img_g3[:, :, 2] = img_g[:, :] image_page, page_coord, cont_page = self.extract_page() - + if self.tables: table_prediction = self.get_tables_from_model(image_page, num_col_classifier) else: table_prediction = np.zeros((image_page.shape[0], image_page.shape[1])).astype(np.int16) - + if self.plotter: self.plotter.save_page_image(image_page) @@ -2818,7 +2868,7 @@ class Eynollah: mask_lines = mask_lines.astype(np.uint8) img_only_regions_with_sep = ((text_regions_p_1[:, :] != 3) & (text_regions_p_1[:, :] != 0)) * 1 img_only_regions_with_sep = img_only_regions_with_sep.astype(np.uint8) - + if erosion_hurts: img_only_regions = np.copy(img_only_regions_with_sep[:,:]) else: @@ -2963,7 +3013,7 @@ class Eynollah: boxes_d = None self.logger.debug("len(boxes): %s", len(boxes)) #print(time.time()-t_0_box,'time box in 3.1') - + if self.tables: if self.light_version: pass @@ -2984,7 +3034,7 @@ class Eynollah: num_col_classifier, erosion_hurts, self.tables, self.right2left) boxes = None self.logger.debug("len(boxes): %s", len(boxes_d)) - + if self.tables: if self.light_version: pass @@ -2992,21 +3042,21 @@ class Eynollah: text_regions_p_tables = np.copy(text_regions_p_1_n) text_regions_p_tables =np.round(text_regions_p_tables) text_regions_p_tables[:,:][(text_regions_p_tables[:,:] != 3) & (table_prediction_n[:,:] == 1)] = 10 - + pixel_line = 3 img_revised_tab2 = self.add_tables_heuristic_to_layout( text_regions_p_tables, boxes_d, 0, splitter_y_new_d, peaks_neg_tot_tables_d, text_regions_p_tables, num_col_classifier, 0.000005, pixel_line) img_revised_tab2_d,_ = self.check_iou_of_bounding_box_and_contour_for_tables( img_revised_tab2, table_prediction_n, 10, num_col_classifier) - + img_revised_tab2_d_rotated = rotate_image(img_revised_tab2_d, -slope_deskew) img_revised_tab2_d_rotated = np.round(img_revised_tab2_d_rotated) img_revised_tab2_d_rotated = img_revised_tab2_d_rotated.astype(np.int8) img_revised_tab2_d_rotated = resize_image(img_revised_tab2_d_rotated, text_regions_p.shape[0], text_regions_p.shape[1]) #print(time.time()-t_0_box,'time box in 4') self.logger.info("detecting boxes took %.1fs", time.time() - t1) - + if self.tables: if self.light_version: text_regions_p[:,:][table_prediction[:,:]==1] = 10 @@ -3019,7 +3069,7 @@ class Eynollah: img_revised_tab = np.copy(text_regions_p[:,:]) img_revised_tab[:,:][img_revised_tab[:,:] == 10] = 0 img_revised_tab[:,:][img_revised_tab2_d_rotated[:,:,0] == 10] = 10 - + text_regions_p[:,:][text_regions_p[:,:]==10] = 0 text_regions_p[:,:][img_revised_tab[:,:]==10] = 10 else: @@ -3036,11 +3086,11 @@ class Eynollah: marginal_mask = (text_regions_p[:,:]==pixel_img)*1 marginal_mask = marginal_mask.astype('uint8') marginal_mask = cv2.dilate(marginal_mask, KERNEL, iterations=2) - + polygons_of_marginals = return_contours_of_interested_region(marginal_mask, 1, min_area_mar) else: polygons_of_marginals = return_contours_of_interested_region(text_regions_p, pixel_img, min_area_mar) - + pixel_img = 10 contours_tables = return_contours_of_interested_region(text_regions_p, pixel_img, min_area_mar) #print(time.time()-t_0_box,'time box in 5') @@ -3063,11 +3113,11 @@ class Eynollah: if np.abs(slope_deskew) >= SLOPE_THRESHOLD: image_page_rotated_n, textline_mask_tot_d, text_regions_p_1_n, table_prediction_n = \ rotation_not_90_func(image_page, textline_mask_tot, text_regions_p, table_prediction, slope_deskew) - + text_regions_p_1_n = resize_image(text_regions_p_1_n,text_regions_p.shape[0],text_regions_p.shape[1]) textline_mask_tot_d = resize_image(textline_mask_tot_d,text_regions_p.shape[0],text_regions_p.shape[1]) table_prediction_n = resize_image(table_prediction_n,text_regions_p.shape[0],text_regions_p.shape[1]) - + regions_without_separators_d = (text_regions_p_1_n[:,:] == 1)*1 regions_without_separators_d[table_prediction_n[:,:] == 1] = 1 else: @@ -3083,11 +3133,11 @@ class Eynollah: if np.abs(slope_deskew) >= SLOPE_THRESHOLD: image_page_rotated_n, textline_mask_tot_d, text_regions_p_1_n, table_prediction_n = \ rotation_not_90_func(image_page, textline_mask_tot, text_regions_p, table_prediction, slope_deskew) - + text_regions_p_1_n = resize_image(text_regions_p_1_n,text_regions_p.shape[0],text_regions_p.shape[1]) textline_mask_tot_d = resize_image(textline_mask_tot_d,text_regions_p.shape[0],text_regions_p.shape[1]) table_prediction_n = resize_image(table_prediction_n,text_regions_p.shape[0],text_regions_p.shape[1]) - + regions_without_separators_d = (text_regions_p_1_n[:,:] == 1)*1 regions_without_separators_d[table_prediction_n[:,:] == 1] = 1 else: @@ -3099,13 +3149,13 @@ class Eynollah: #self.return_regions_without_separators_new(text_regions_p[:,:,0],img_only_regions) regions_without_separators = (text_regions_p[:,:] == 1)*1 regions_without_separators[table_prediction == 1] = 1 - + pixel_lines=3 if np.abs(slope_deskew) < SLOPE_THRESHOLD: num_col, _, matrix_of_lines_ch, splitter_y_new, _ = find_number_of_columns_in_document( np.repeat(text_regions_p[:, :, np.newaxis], 3, axis=2), num_col_classifier, self.tables, pixel_lines) - + if np.abs(slope_deskew) >= SLOPE_THRESHOLD: num_col_d, _, matrix_of_lines_ch_d, splitter_y_new_d, _ = find_number_of_columns_in_document( np.repeat(text_regions_p_1_n[:, :, np.newaxis], 3, axis=2), @@ -3115,13 +3165,13 @@ class Eynollah: if np.abs(slope_deskew) < SLOPE_THRESHOLD: regions_without_separators = regions_without_separators.astype(np.uint8) regions_without_separators = cv2.erode(regions_without_separators[:,:], KERNEL, iterations=6) - + if np.abs(slope_deskew) >= SLOPE_THRESHOLD: regions_without_separators_d = regions_without_separators_d.astype(np.uint8) regions_without_separators_d = cv2.erode(regions_without_separators_d[:,:], KERNEL, iterations=6) else: pass - + if np.abs(slope_deskew) < SLOPE_THRESHOLD: boxes, peaks_neg_tot_tables = return_boxes_of_images_by_order_of_reading_new( splitter_y_new, regions_without_separators, matrix_of_lines_ch, @@ -3132,7 +3182,7 @@ class Eynollah: img_revised_tab2 = self.add_tables_heuristic_to_layout( text_regions_p_tables, boxes, 0, splitter_y_new, peaks_neg_tot_tables, text_regions_p_tables, num_col_classifier , 0.000005, pixel_line) - + img_revised_tab2,contoures_tables = self.check_iou_of_bounding_box_and_contour_for_tables( img_revised_tab2, table_prediction, 10, num_col_classifier) else: @@ -3142,12 +3192,12 @@ class Eynollah: text_regions_p_tables = np.copy(text_regions_p_1_n) text_regions_p_tables = np.round(text_regions_p_tables) text_regions_p_tables[:,:][(text_regions_p_tables[:,:]!=3) & (table_prediction_n[:,:]==1)] = 10 - + pixel_line = 3 img_revised_tab2 = self.add_tables_heuristic_to_layout( text_regions_p_tables, boxes_d, 0, splitter_y_new_d, peaks_neg_tot_tables_d, text_regions_p_tables, num_col_classifier, 0.000005, pixel_line) - + img_revised_tab2_d,_ = self.check_iou_of_bounding_box_and_contour_for_tables( img_revised_tab2, table_prediction_n, 10, num_col_classifier) img_revised_tab2_d_rotated = rotate_image(img_revised_tab2_d, -slope_deskew) @@ -3163,28 +3213,28 @@ class Eynollah: img_revised_tab = np.copy(text_regions_p[:,:]) img_revised_tab[:,:][img_revised_tab[:,:] == 10] = 0 img_revised_tab[:,:][img_revised_tab2_d_rotated[:,:,0] == 10] = 10 - + ##img_revised_tab=img_revised_tab2[:,:,0] #img_revised_tab=text_regions_p[:,:] text_regions_p[:,:][text_regions_p[:,:]==10] = 0 text_regions_p[:,:][img_revised_tab[:,:]==10] = 10 #img_revised_tab[img_revised_tab2[:,:,0]==10] =10 - + pixel_img = 4 min_area_mar = 0.00001 - + if self.light_version: marginal_mask = (text_regions_p[:,:]==pixel_img)*1 marginal_mask = marginal_mask.astype('uint8') marginal_mask = cv2.dilate(marginal_mask, KERNEL, iterations=2) - + polygons_of_marginals = return_contours_of_interested_region(marginal_mask, 1, min_area_mar) else: polygons_of_marginals = return_contours_of_interested_region(text_regions_p, pixel_img, min_area_mar) - + pixel_img = 10 contours_tables = return_contours_of_interested_region(text_regions_p, pixel_img, min_area_mar) - + # set first model with second model text_regions_p[:, :][text_regions_p[:, :] == 2] = 5 text_regions_p[:, :][text_regions_p[:, :] == 3] = 6 @@ -3199,23 +3249,23 @@ class Eynollah: # 6 is the separators lable in old full layout model # 4 is the drop capital class in old full layout model # in the new full layout drop capital is 3 and separators are 5 - + text_regions_p[:,:][regions_fully[:,:,0]==5]=6 ###regions_fully[:, :, 0][regions_fully_only_drop[:, :, 0] == 3] = 4 - + #text_regions_p[:,:][regions_fully[:,:,0]==6]=6 ##regions_fully_only_drop = put_drop_out_from_only_drop_model(regions_fully_only_drop, text_regions_p) ##regions_fully[:, :, 0][regions_fully_only_drop[:, :, 0] == 4] = 4 drop_capital_label_in_full_layout_model = 3 - + drops = (regions_fully[:,:,0]==drop_capital_label_in_full_layout_model)*1 drops= drops.astype(np.uint8) - + regions_fully[:,:,0][regions_fully[:,:,0]==drop_capital_label_in_full_layout_model] = 1 - + drops = cv2.erode(drops[:,:], KERNEL, iterations=1) regions_fully[:,:,0][drops[:,:]==1] = drop_capital_label_in_full_layout_model - + regions_fully = putt_bb_of_drop_capitals_of_model_in_patches_in_layout( regions_fully, drop_capital_label_in_full_layout_model, text_regions_p) ##regions_fully_np, _ = self.extract_text_regions(image_page, False, cols=num_col_classifier) @@ -3249,7 +3299,7 @@ class Eynollah: regions_without_separators = (text_regions_p[:, :] == 1) * 1 img_revised_tab = np.copy(text_regions_p[:, :]) polygons_of_images = return_contours_of_interested_region(img_revised_tab, 5) - + self.logger.debug('exit run_boxes_full_layout') #print("full inside 3", time.time()- t_full0) return (polygons_of_images, img_revised_tab, text_regions_p_1_n, textline_mask_tot_d, @@ -3371,10 +3421,10 @@ class Eynollah: width1 = int ( width/2. - common_window ) width2 = int ( width/2. + common_window ) - + img_sum = np.sum(textline_image[:,:,0], axis=0) sum_smoothed = gaussian_filter1d(img_sum, 3) - + peaks_real, _ = find_peaks(sum_smoothed, height=0) if len(peaks_real)>70: print(len(peaks_real), 'len(peaks_real)') @@ -3389,16 +3439,16 @@ class Eynollah: first_4_sorted = peaks_sort_4[argsort_sorted] y_4_sorted = sum_smoothed[peaks_real][arg_sort4[argsort_sorted]] #print(first_4_sorted,'first_4_sorted') - + arg_sortnew = np.argsort(y_4_sorted) peaks_final =np.sort( first_4_sorted[arg_sortnew][2:] ) - + #plt.figure(ind_tot) #plt.imshow(textline_image) #plt.plot([peaks_final[0], peaks_final[0]], [0, height-1]) #plt.plot([peaks_final[1], peaks_final[1]], [0, height-1]) #plt.savefig('./'+str(ind_tot)+'.png') - + return peaks_final[0], peaks_final[1] else: pass @@ -3410,10 +3460,10 @@ class Eynollah: width1 = int ( width/2. - common_window ) width2 = int ( width/2. + common_window ) - + img_sum = np.sum(textline_image[:,:,0], axis=0) sum_smoothed = gaussian_filter1d(img_sum, 3) - + peaks_real, _ = find_peaks(sum_smoothed, height=0) if len(peaks_real)>70: #print(len(peaks_real), 'len(peaks_real)') @@ -3422,13 +3472,13 @@ class Eynollah: arg_max = np.argmax(sum_smoothed[peaks_real]) peaks_final = peaks_real[arg_max] - + #plt.figure(ind_tot) #plt.imshow(textline_image) #plt.plot([peaks_final, peaks_final], [0, height-1]) ##plt.plot([peaks_final[1], peaks_final[1]], [0, height-1]) #plt.savefig('./'+str(ind_tot)+'.png') - + return peaks_final else: return None @@ -3446,7 +3496,7 @@ class Eynollah: first_4_sorted = peaks_sort_4[argsort_sorted] y_4_sorted = sum_smoothed[peaks_real][arg_sort4[argsort_sorted]] #print(first_4_sorted,'first_4_sorted') - + arg_sortnew = np.argsort(y_4_sorted) peaks_final =np.sort( first_4_sorted[arg_sortnew][3:] ) return peaks_final[0] @@ -3459,23 +3509,23 @@ class Eynollah: width1 = int ( width/2. - common_window ) width2 = int ( width/2. + common_window ) mid = int(width/2.) - + img_sum = np.sum(textline_image[:,:,0], axis=0) sum_smoothed = gaussian_filter1d(img_sum, 3) - + peaks_real, _ = find_peaks(sum_smoothed, height=0) if len(peaks_real)>70: peak_start = self.return_start_and_end_of_common_text_of_textline_ocr_new_splitted( peaks_real, sum_smoothed, width1, mid+2) peak_end = self.return_start_and_end_of_common_text_of_textline_ocr_new_splitted( peaks_real, sum_smoothed, mid-2, width2) - + #plt.figure(ind_tot) #plt.imshow(textline_image) #plt.plot([peak_start, peak_start], [0, height-1]) #plt.plot([peak_end, peak_end], [0, height-1]) #plt.savefig('./'+str(ind_tot)+'.png') - + return peak_start, peak_end else: pass @@ -3493,31 +3543,31 @@ class Eynollah: #common_window = int(0.3*width) #width1 = int ( width/2. - common_window ) #width2 = int ( width/2. + common_window ) - + split_point = self.return_start_and_end_of_common_text_of_textline_ocr_without_common_section( textline_image, ind_tot) if split_point: image1 = textline_image[:, :split_point,:]# image.crop((0, 0, width2, height)) image2 = textline_image[:, split_point:,:]#image.crop((width1, 0, width, height)) - + #pixel_values1 = processor(image1, return_tensors="pt").pixel_values #pixel_values2 = processor(image2, return_tensors="pt").pixel_values - + pixel_values_merged = processor([image1,image2], return_tensors="pt").pixel_values generated_ids_merged = model_ocr.generate(pixel_values_merged.to(device)) generated_text_merged = processor.batch_decode(generated_ids_merged, skip_special_tokens=True) - + #print(generated_text_merged,'generated_text_merged') - + #generated_ids1 = model_ocr.generate(pixel_values1.to(device)) #generated_ids2 = model_ocr.generate(pixel_values2.to(device)) - + #generated_text1 = processor.batch_decode(generated_ids1, skip_special_tokens=True)[0] #generated_text2 = processor.batch_decode(generated_ids2, skip_special_tokens=True)[0] - + #generated_text = generated_text1 + ' ' + generated_text2 generated_text = generated_text_merged[0] + ' ' + generated_text_merged[1] - + #print(generated_text1,'generated_text1') #print(generated_text2, 'generated_text2') #print('########################################') @@ -3525,7 +3575,7 @@ class Eynollah: pixel_values = processor(textline_image, return_tensors="pt").pixel_values generated_ids = model_ocr.generate(pixel_values.to(device)) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] - + #print(generated_text,'generated_text') #print('########################################') return generated_text @@ -3543,25 +3593,25 @@ class Eynollah: #common_window = int(0.3*width) #width1 = int ( width/2. - common_window ) #width2 = int ( width/2. + common_window ) - + try: width1, width2 = self.return_start_and_end_of_common_text_of_textline_ocr_new(textline_image, ind_tot) - + image1 = textline_image[:, :width2,:]# image.crop((0, 0, width2, height)) image2 = textline_image[:, width1:,:]#image.crop((width1, 0, width, height)) - + pixel_values1 = processor(image1, return_tensors="pt").pixel_values pixel_values2 = processor(image2, return_tensors="pt").pixel_values - + generated_ids1 = model_ocr.generate(pixel_values1.to(device)) generated_ids2 = model_ocr.generate(pixel_values2.to(device)) - + generated_text1 = processor.batch_decode(generated_ids1, skip_special_tokens=True)[0] generated_text2 = processor.batch_decode(generated_ids2, skip_special_tokens=True)[0] #print(generated_text1,'generated_text1') #print(generated_text2, 'generated_text2') #print('########################################') - + match = sq(None, generated_text1, generated_text2).find_longest_match( 0, len(generated_text1), 0, len(generated_text2)) generated_text = generated_text1 + generated_text2[match.b+match.size:] @@ -3569,7 +3619,7 @@ class Eynollah: pixel_values = processor(textline_image, return_tensors="pt").pixel_values generated_ids = model_ocr.generate(pixel_values.to(device)) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] - + return generated_text def return_textline_contour_with_added_box_coordinate(self, textline_contour, box_ind): @@ -3602,7 +3652,7 @@ class Eynollah: for i in indexes_to_bec_changed_filtered: i_slice = slice(split_masked[i], split_masked[i+1]) x_differential_new[i_slice] = -1 * np.array(x_differential)[i_slice] - + return x_differential_new def dilate_textregions_contours_textline_version(self, all_found_textline_polygons): @@ -3612,32 +3662,32 @@ class Eynollah: con_ind = all_found_textline_polygons[j][ij] area = cv2.contourArea(con_ind) con_ind = con_ind.astype(float) - + x_differential = np.diff( con_ind[:,0,0]) y_differential = np.diff( con_ind[:,0,1]) - + x_differential = gaussian_filter1d(x_differential, 0.1) y_differential = gaussian_filter1d(y_differential, 0.1) - + x_min = float(np.min( con_ind[:,0,0] )) y_min = float(np.min( con_ind[:,0,1] )) - + x_max = float(np.max( con_ind[:,0,0] )) y_max = float(np.max( con_ind[:,0,1] )) - + x_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in x_differential] y_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in y_differential] - + abs_diff=abs(abs(x_differential)- abs(y_differential) ) - + inc_x = np.zeros(len(x_differential)+1) inc_y = np.zeros(len(x_differential)+1) - + if (y_max-y_min) <= (x_max-x_min): dilation_m1 = round(area / (x_max-x_min) * 0.12) else: dilation_m1 = round(area / (y_max-y_min) * 0.12) - + if dilation_m1>8: dilation_m1 = 8 if dilation_m1<6: @@ -3645,7 +3695,7 @@ class Eynollah: #print(dilation_m1, 'dilation_m1') dilation_m1 = 6 dilation_m2 = int(dilation_m1/2.) +1 - + for i in range(len(x_differential)): if abs_diff[i]==0: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) @@ -3654,7 +3704,7 @@ class Eynollah: inc_x[i+1]= dilation_m1*(-1*y_differential_mask_nonzeros[i]) elif abs_diff[i]!=0 and x_differential_mask_nonzeros[i]!=0 and y_differential_mask_nonzeros[i]==0: inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) - + elif abs_diff[i]!=0 and abs_diff[i]>=3: if abs(x_differential[i])>abs(y_differential[i]): inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) @@ -3663,44 +3713,44 @@ class Eynollah: else: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) inc_y[i+1] = dilation_m2*(x_differential_mask_nonzeros[i]) - + inc_x[0] = inc_x[-1] inc_y[0] = inc_y[-1] - + con_scaled = con_ind*1 - + con_scaled[:,0, 0] = con_ind[:,0,0] + np.array(inc_x)[:] con_scaled[:,0, 1] = con_ind[:,0,1] + np.array(inc_y)[:] - + con_scaled[:,0, 1][con_scaled[:,0, 1]<0] = 0 con_scaled[:,0, 0][con_scaled[:,0, 0]<0] = 0 - + area_scaled = cv2.contourArea(con_scaled.astype(np.int32)) - + con_ind = con_ind.astype(np.int32) - + results = [cv2.pointPolygonTest(con_ind, (con_scaled[ind,0, 0], con_scaled[ind,0, 1]), False) for ind in range(len(con_scaled[:,0, 1])) ] results = np.array(results) #print(results,'results') results[results==0] = 1 - + diff_result = np.diff(results) - + indices_2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==2] indices_m2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==-2] - + if results[0]==1: con_scaled[:indices_m2[0]+1,0, 1] = con_ind[:indices_m2[0]+1,0,1] con_scaled[:indices_m2[0]+1,0, 0] = con_ind[:indices_m2[0]+1,0,0] #indices_2 = indices_2[1:] indices_m2 = indices_m2[1:] - + if len(indices_2)>len(indices_m2): con_scaled[indices_2[-1]+1:,0, 1] = con_ind[indices_2[-1]+1:,0,1] con_scaled[indices_2[-1]+1:,0, 0] = con_ind[indices_2[-1]+1:,0,0] indices_2 = indices_2[:-1] - + for ii in range(len(indices_2)): con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 1] = con_scaled[indices_2[ii],0, 1] con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 0] = con_scaled[indices_2[ii],0, 0] @@ -3716,32 +3766,32 @@ class Eynollah: #print(len(con_ind[:,0,0]),'con_ind[:,0,0]') area = cv2.contourArea(con_ind) con_ind = con_ind.astype(float) - + x_differential = np.diff( con_ind[:,0,0]) y_differential = np.diff( con_ind[:,0,1]) - + x_differential = gaussian_filter1d(x_differential, 0.1) y_differential = gaussian_filter1d(y_differential, 0.1) - + x_min = float(np.min( con_ind[:,0,0] )) y_min = float(np.min( con_ind[:,0,1] )) - + x_max = float(np.max( con_ind[:,0,0] )) y_max = float(np.max( con_ind[:,0,1] )) - + x_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in x_differential] y_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in y_differential] - + abs_diff=abs(abs(x_differential)- abs(y_differential) ) - + inc_x = np.zeros(len(x_differential)+1) inc_y = np.zeros(len(x_differential)+1) - + if (y_max-y_min) <= (x_max-x_min): dilation_m1 = round(area / (x_max-x_min) * 0.12) else: dilation_m1 = round(area / (y_max-y_min) * 0.12) - + if dilation_m1>8: dilation_m1 = 8 if dilation_m1<6: @@ -3749,7 +3799,7 @@ class Eynollah: #print(dilation_m1, 'dilation_m1') dilation_m1 = 6 dilation_m2 = int(dilation_m1/2.) +1 - + for i in range(len(x_differential)): if abs_diff[i]==0: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) @@ -3758,7 +3808,7 @@ class Eynollah: inc_x[i+1]= dilation_m1*(-1*y_differential_mask_nonzeros[i]) elif abs_diff[i]!=0 and x_differential_mask_nonzeros[i]!=0 and y_differential_mask_nonzeros[i]==0: inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) - + elif abs_diff[i]!=0 and abs_diff[i]>=3: if abs(x_differential[i])>abs(y_differential[i]): inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) @@ -3767,28 +3817,28 @@ class Eynollah: else: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) inc_y[i+1] = dilation_m2*(x_differential_mask_nonzeros[i]) - + inc_x[0] = inc_x[-1] inc_y[0] = inc_y[-1] - + con_scaled = con_ind*1 - + con_scaled[:,0, 0] = con_ind[:,0,0] + np.array(inc_x)[:] con_scaled[:,0, 1] = con_ind[:,0,1] + np.array(inc_y)[:] - + con_scaled[:,0, 1][con_scaled[:,0, 1]<0] = 0 con_scaled[:,0, 0][con_scaled[:,0, 0]<0] = 0 - + area_scaled = cv2.contourArea(con_scaled.astype(np.int32)) - + con_ind = con_ind.astype(np.int32) - + results = [cv2.pointPolygonTest(con_ind, (con_scaled[ind,0, 0], con_scaled[ind,0, 1]), False) for ind in range(len(con_scaled[:,0, 1])) ] results = np.array(results) #print(results,'results') results[results==0] = 1 - + diff_result = np.diff(results) indices_2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==2] indices_m2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==-2] @@ -3798,12 +3848,12 @@ class Eynollah: con_scaled[:indices_m2[0]+1,0, 0] = con_ind[:indices_m2[0]+1,0,0] #indices_2 = indices_2[1:] indices_m2 = indices_m2[1:] - + if len(indices_2)>len(indices_m2): con_scaled[indices_2[-1]+1:,0, 1] = con_ind[indices_2[-1]+1:,0,1] con_scaled[indices_2[-1]+1:,0, 0] = con_ind[indices_2[-1]+1:,0,0] indices_2 = indices_2[:-1] - + for ii in range(len(indices_2)): con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 1] = con_scaled[indices_2[ii],0, 1] con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 0] = con_scaled[indices_2[ii],0, 0] @@ -3811,47 +3861,47 @@ class Eynollah: all_found_textline_polygons[j][:,0,1] = con_scaled[:,0, 1] all_found_textline_polygons[j][:,0,0] = con_scaled[:,0, 0] return all_found_textline_polygons - + def dilate_textline_contours(self, all_found_textline_polygons): for j in range(len(all_found_textline_polygons)): for ij in range(len(all_found_textline_polygons[j])): con_ind = all_found_textline_polygons[j][ij] area = cv2.contourArea(con_ind) - + con_ind = con_ind.astype(float) - + x_differential = np.diff( con_ind[:,0,0]) y_differential = np.diff( con_ind[:,0,1]) - + x_differential = gaussian_filter1d(x_differential, 3) y_differential = gaussian_filter1d(y_differential, 3) - + x_min = float(np.min( con_ind[:,0,0] )) y_min = float(np.min( con_ind[:,0,1] )) - + x_max = float(np.max( con_ind[:,0,0] )) y_max = float(np.max( con_ind[:,0,1] )) - + x_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in x_differential] y_differential_mask_nonzeros = [ ind/abs(ind) if ind!=0 else ind for ind in y_differential] - + abs_diff=abs(abs(x_differential)- abs(y_differential) ) - + inc_x = np.zeros(len(x_differential)+1) inc_y = np.zeros(len(x_differential)+1) - + if (y_max-y_min) <= (x_max-x_min): dilation_m1 = round(area / (x_max-x_min) * 0.35) else: dilation_m1 = round(area / (y_max-y_min) * 0.35) - + if dilation_m1>12: dilation_m1 = 12 if dilation_m1<4: dilation_m1 = 4 #print(dilation_m1, 'dilation_m1') dilation_m2 = int(dilation_m1/2.) +1 - + for i in range(len(x_differential)): if abs_diff[i]==0: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) @@ -3860,7 +3910,7 @@ class Eynollah: inc_x[i+1]= dilation_m1*(-1*y_differential_mask_nonzeros[i]) elif abs_diff[i]!=0 and x_differential_mask_nonzeros[i]!=0 and y_differential_mask_nonzeros[i]==0: inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) - + elif abs_diff[i]!=0 and abs_diff[i]>=3: if abs(x_differential[i])>abs(y_differential[i]): inc_y[i+1] = dilation_m1*(x_differential_mask_nonzeros[i]) @@ -3869,62 +3919,62 @@ class Eynollah: else: inc_x[i+1] = dilation_m2*(-1*y_differential_mask_nonzeros[i]) inc_y[i+1] = dilation_m2*(x_differential_mask_nonzeros[i]) - + inc_x[0] = inc_x[-1] inc_y[0] = inc_y[-1] - + con_scaled = con_ind*1 - + con_scaled[:,0, 0] = con_ind[:,0,0] + np.array(inc_x)[:] con_scaled[:,0, 1] = con_ind[:,0,1] + np.array(inc_y)[:] - + con_scaled[:,0, 1][con_scaled[:,0, 1]<0] = 0 con_scaled[:,0, 0][con_scaled[:,0, 0]<0] = 0 - + con_ind = con_ind.astype(np.int32) - + results = [cv2.pointPolygonTest(con_ind, (con_scaled[ind,0, 0], con_scaled[ind,0, 1]), False) for ind in range(len(con_scaled[:,0, 1])) ] results = np.array(results) results[results==0] = 1 - + diff_result = np.diff(results) - + indices_2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==2] indices_m2 = [ind for ind in range(len(diff_result)) if diff_result[ind]==-2] - + if results[0]==1: con_scaled[:indices_m2[0]+1,0, 1] = con_ind[:indices_m2[0]+1,0,1] con_scaled[:indices_m2[0]+1,0, 0] = con_ind[:indices_m2[0]+1,0,0] indices_m2 = indices_m2[1:] - + if len(indices_2)>len(indices_m2): con_scaled[indices_2[-1]+1:,0, 1] = con_ind[indices_2[-1]+1:,0,1] con_scaled[indices_2[-1]+1:,0, 0] = con_ind[indices_2[-1]+1:,0,0] indices_2 = indices_2[:-1] - + for ii in range(len(indices_2)): con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 1] = con_scaled[indices_2[ii],0, 1] con_scaled[indices_2[ii]+1:indices_m2[ii]+1,0, 0] = con_scaled[indices_2[ii],0, 0] - + all_found_textline_polygons[j][ij][:,0,1] = con_scaled[:,0, 1] all_found_textline_polygons[j][ij][:,0,0] = con_scaled[:,0, 0] return all_found_textline_polygons - - def filter_contours_inside_a_bigger_one(self,contours, image, marginal_cnts=None, type_contour="textregion"): + + def filter_contours_inside_a_bigger_one(self,contours, contours_d_ordered, image, marginal_cnts=None, type_contour="textregion"): if type_contour=="textregion": areas = [cv2.contourArea(contours[j]) for j in range(len(contours))] area_tot = image.shape[0]*image.shape[1] - + M_main = [cv2.moments(contours[j]) for j in range(len(contours))] cx_main = [(M_main[j]["m10"] / (M_main[j]["m00"] + 1e-32)) for j in range(len(M_main))] cy_main = [(M_main[j]["m01"] / (M_main[j]["m00"] + 1e-32)) for j in range(len(M_main))] - + areas_ratio = np.array(areas)/ area_tot contours_index_small = [ind for ind in range(len(contours)) if areas_ratio[ind] < 1e-3] contours_index_big = [ind for ind in range(len(contours)) if areas_ratio[ind] >= 1e-3] - + #contours_> = [contours[ind] for ind in contours_index_big] indexes_to_be_removed = [] for ind_small in contours_index_small: @@ -3934,55 +3984,57 @@ class Eynollah: results_marginal = [cv2.pointPolygonTest(marginal_cnts[ind], (cx_main[ind_small], cy_main[ind_small]), False) for ind in range(len(marginal_cnts))] results_marginal = np.array(results_marginal) - + if np.any(results_marginal==1): indexes_to_be_removed.append(ind_small) - + results = np.array(results) - + if np.any(results==1): indexes_to_be_removed.append(ind_small) - + if len(indexes_to_be_removed)>0: indexes_to_be_removed = np.unique(indexes_to_be_removed) indexes_to_be_removed = np.sort(indexes_to_be_removed)[::-1] for ind in indexes_to_be_removed: contours.pop(ind) + if len(contours_d_ordered)>0: + contours_d_ordered.pop(ind) - return contours + return contours, contours_d_ordered else: contours_txtline_of_all_textregions = [] indexes_of_textline_tot = [] index_textline_inside_textregion = [] - + for jj in range(len(contours)): contours_txtline_of_all_textregions = contours_txtline_of_all_textregions + contours[jj] - + ind_textline_inside_tr = list(range(len(contours[jj]))) index_textline_inside_textregion = index_textline_inside_textregion + ind_textline_inside_tr ind_ins = [jj] * len(contours[jj]) indexes_of_textline_tot = indexes_of_textline_tot + ind_ins - + M_main_tot = [cv2.moments(contours_txtline_of_all_textregions[j]) for j in range(len(contours_txtline_of_all_textregions))] cx_main_tot = [(M_main_tot[j]["m10"] / (M_main_tot[j]["m00"] + 1e-32)) for j in range(len(M_main_tot))] cy_main_tot = [(M_main_tot[j]["m01"] / (M_main_tot[j]["m00"] + 1e-32)) for j in range(len(M_main_tot))] - + areas_tot = [cv2.contourArea(con_ind) for con_ind in contours_txtline_of_all_textregions] area_tot_tot = image.shape[0]*image.shape[1] - + textregion_index_to_del = [] textline_in_textregion_index_to_del = [] for ij in range(len(contours_txtline_of_all_textregions)): args_all = list(np.array(range(len(contours_txtline_of_all_textregions)))) args_all.pop(ij) - + areas_without = np.array(areas_tot)[args_all] area_of_con_interest = areas_tot[ij] - + args_with_bigger_area = np.array(args_all)[areas_without > 1.5*area_of_con_interest] - + if len(args_with_bigger_area)>0: results = [cv2.pointPolygonTest(contours_txtline_of_all_textregions[ind], (cx_main_tot[ij], cy_main_tot[ij]), False) for ind in args_with_bigger_area ] @@ -4000,27 +4052,26 @@ class Eynollah: textline_in_textregion_index_to_del_ind = np.sort(textline_in_textregion_index_to_del_ind)[::-1] for ittrd in textline_in_textregion_index_to_del_ind: contours[ind_u_a_trs].pop(ittrd) - + return contours - + def filter_contours_without_textline_inside( - self, contours,text_con_org, contours_textline, contours_only_text_parent_d_ordered): - + self, contours,text_con_org, contours_textline, contours_only_text_parent_d_ordered, conf_contours_textregions): ###contours_txtline_of_all_textregions = [] ###for jj in range(len(contours_textline)): ###contours_txtline_of_all_textregions = contours_txtline_of_all_textregions + contours_textline[jj] - + ###M_main_textline = [cv2.moments(contours_txtline_of_all_textregions[j]) ### for j in range(len(contours_txtline_of_all_textregions))] ###cx_main_textline = [(M_main_textline[j]["m10"] / (M_main_textline[j]["m00"] + 1e-32)) ### for j in range(len(M_main_textline))] ###cy_main_textline = [(M_main_textline[j]["m01"] / (M_main_textline[j]["m00"] + 1e-32)) ### for j in range(len(M_main_textline))] - + ###M_main = [cv2.moments(contours[j]) for j in range(len(contours))] ###cx_main = [(M_main[j]["m10"] / (M_main[j]["m00"] + 1e-32)) for j in range(len(M_main))] ###cy_main = [(M_main[j]["m01"] / (M_main[j]["m00"] + 1e-32)) for j in range(len(M_main))] - + ###contours_with_textline = [] ###for ind_tr, con_tr in enumerate(contours): ###results = [cv2.pointPolygonTest(con_tr, (cx_main_textline[index_textline_con], cy_main_textline[index_textline_con]), False) @@ -4028,7 +4079,7 @@ class Eynollah: ###results = np.array(results) ###if np.any(results==1): ###contours_with_textline.append(con_tr) - + textregion_index_to_del = [] for index_textregion, textlines_textregion in enumerate(contours_textline): if len(textlines_textregion)==0: @@ -4036,38 +4087,39 @@ class Eynollah: uniqe_args_trs = np.unique(textregion_index_to_del) uniqe_args_trs_sorted = np.sort(uniqe_args_trs)[::-1] - + for ind_u_a_trs in uniqe_args_trs_sorted: + conf_contours_textregions.pop(ind_u_a_trs) contours.pop(ind_u_a_trs) contours_textline.pop(ind_u_a_trs) text_con_org.pop(ind_u_a_trs) if len(contours_only_text_parent_d_ordered) > 0: contours_only_text_parent_d_ordered.pop(ind_u_a_trs) - - return contours, text_con_org, contours_textline, contours_only_text_parent_d_ordered, np.array(range(len(contours))) - + + return contours, text_con_org, conf_contours_textregions, contours_textline, contours_only_text_parent_d_ordered, np.array(range(len(contours))) + def dilate_textlines(self, all_found_textline_polygons): for j in range(len(all_found_textline_polygons)): for i in range(len(all_found_textline_polygons[j])): con_ind = all_found_textline_polygons[j][i] con_ind = con_ind.astype(float) - + x_differential = np.diff( con_ind[:,0,0]) y_differential = np.diff( con_ind[:,0,1]) - + x_min = float(np.min( con_ind[:,0,0] )) y_min = float(np.min( con_ind[:,0,1] )) - + x_max = float(np.max( con_ind[:,0,0] )) y_max = float(np.max( con_ind[:,0,1] )) if (y_max - y_min) > (x_max - x_min) and (x_max - x_min)<70: x_biger_than_x = np.abs(x_differential) > np.abs(y_differential) mult = x_biger_than_x*x_differential - + arg_min_mult = np.argmin(mult) arg_max_mult = np.argmax(mult) - + if y_differential[0]==0: y_differential[0] = 0.1 if y_differential[-1]==0: @@ -4075,45 +4127,45 @@ class Eynollah: y_differential = [y_differential[ind] if y_differential[ind] != 0 else 0.5 * (y_differential[ind-1] + y_differential[ind+1]) for ind in range(len(y_differential))] - + if y_differential[0]==0.1: y_differential[0] = y_differential[1] if y_differential[-1]==0.1: y_differential[-1] = y_differential[-2] y_differential.append(y_differential[0]) - + y_differential = [-1 if y_differential[ind] < 0 else 1 for ind in range(len(y_differential))] y_differential = self.return_it_in_two_groups(y_differential) y_differential = np.array(y_differential) - + con_scaled = con_ind*1 con_scaled[:,0, 0] = con_ind[:,0,0] - 8*y_differential con_scaled[arg_min_mult,0, 1] = con_ind[arg_min_mult,0,1] + 8 con_scaled[arg_min_mult+1,0, 1] = con_ind[arg_min_mult+1,0,1] + 8 - + try: con_scaled[arg_min_mult-1,0, 1] = con_ind[arg_min_mult-1,0,1] + 5 con_scaled[arg_min_mult+2,0, 1] = con_ind[arg_min_mult+2,0,1] + 5 except: pass - + con_scaled[arg_max_mult,0, 1] = con_ind[arg_max_mult,0,1] - 8 con_scaled[arg_max_mult+1,0, 1] = con_ind[arg_max_mult+1,0,1] - 8 - + try: con_scaled[arg_max_mult-1,0, 1] = con_ind[arg_max_mult-1,0,1] - 5 con_scaled[arg_max_mult+2,0, 1] = con_ind[arg_max_mult+2,0,1] - 5 except: pass - + else: y_biger_than_x = np.abs(y_differential) > np.abs(x_differential) mult = y_biger_than_x*y_differential - + arg_min_mult = np.argmin(mult) arg_max_mult = np.argmax(mult) - + if x_differential[0]==0: x_differential[0] = 0.1 if x_differential[-1]==0: @@ -4121,32 +4173,32 @@ class Eynollah: x_differential = [x_differential[ind] if x_differential[ind] != 0 else 0.5 * (x_differential[ind-1] + x_differential[ind+1]) for ind in range(len(x_differential))] - + if x_differential[0]==0.1: x_differential[0] = x_differential[1] if x_differential[-1]==0.1: x_differential[-1] = x_differential[-2] x_differential.append(x_differential[0]) - + x_differential = [-1 if x_differential[ind] < 0 else 1 for ind in range(len(x_differential))] x_differential = self.return_it_in_two_groups(x_differential) x_differential = np.array(x_differential) - + con_scaled = con_ind*1 con_scaled[:,0, 1] = con_ind[:,0,1] + 8*x_differential con_scaled[arg_min_mult,0, 0] = con_ind[arg_min_mult,0,0] + 8 con_scaled[arg_min_mult+1,0, 0] = con_ind[arg_min_mult+1,0,0] + 8 - + try: con_scaled[arg_min_mult-1,0, 0] = con_ind[arg_min_mult-1,0,0] + 5 con_scaled[arg_min_mult+2,0, 0] = con_ind[arg_min_mult+2,0,0] + 5 except: pass - + con_scaled[arg_max_mult,0, 0] = con_ind[arg_max_mult,0,0] - 8 con_scaled[arg_max_mult+1,0, 0] = con_ind[arg_max_mult+1,0,0] - 8 - + try: con_scaled[arg_max_mult-1,0, 0] = con_ind[arg_max_mult-1,0,0] - 5 con_scaled[arg_max_mult+2,0, 0] = con_ind[arg_max_mult+2,0,0] - 5 @@ -4155,12 +4207,12 @@ class Eynollah: con_scaled[:,0, 1][con_scaled[:,0, 1]<0] = 0 con_scaled[:,0, 0][con_scaled[:,0, 0]<0] = 0 - + all_found_textline_polygons[j][i][:,0,1] = con_scaled[:,0, 1] all_found_textline_polygons[j][i][:,0,0] = con_scaled[:,0, 0] return all_found_textline_polygons - + def delete_regions_without_textlines( self, slopes, all_found_textline_polygons, boxes_text, txt_con_org, contours_only_text_parent, index_by_text_par_con): @@ -4171,7 +4223,7 @@ class Eynollah: txt_con_org_rem = [] contours_only_text_parent_rem = [] index_by_text_par_con_rem = [] - + for i, ind_con in enumerate(all_found_textline_polygons): if len(ind_con): all_found_textline_polygons_rem.append(ind_con) @@ -4180,13 +4232,13 @@ class Eynollah: txt_con_org_rem.append(txt_con_org[i]) contours_only_text_parent_rem.append(contours_only_text_parent[i]) index_by_text_par_con_rem.append(index_by_text_par_con[i]) - + index_sort = np.argsort(index_by_text_par_con_rem) indexes_new = np.array(range(len(index_by_text_par_con_rem))) - + index_by_text_par_con_rem_sort = [indexes_new[index_sort==j][0] for j in range(len(index_by_text_par_con_rem))] - + return (slopes_rem, all_found_textline_polygons_rem, boxes_text_rem, txt_con_org_rem, contours_only_text_parent_rem, index_by_text_par_con_rem_sort) @@ -4237,13 +4289,13 @@ class Eynollah: pcgts = self.writer.build_pagexml_no_full_layout( [], page_coord, [], [], [], [], polygons_of_images, [], [], [], [], [], - cont_page, [], [], ocr_all_textlines) + cont_page, [], [], ocr_all_textlines, []) if self.plotter: self.plotter.write_images_into_directory(polygons_of_images, image_page) return pcgts if self.skip_layout_and_reading_order: - _ ,_, _, textline_mask_tot_ea, img_bin_light = \ + _ ,_, _, textline_mask_tot_ea, img_bin_light, _ = \ self.get_regions_light_v(img_res, is_image_enhanced, num_col_classifier, skip_layout_and_reading_order=self.skip_layout_and_reading_order) @@ -4262,7 +4314,7 @@ class Eynollah: all_found_textline_polygons = self.dilate_textregions_contours_textline_version( all_found_textline_polygons) all_found_textline_polygons = self.filter_contours_inside_a_bigger_one( - all_found_textline_polygons, textline_mask_tot_ea, type_contour="textline") + all_found_textline_polygons, None, textline_mask_tot_ea, type_contour="textline") order_text_new = [0] @@ -4277,17 +4329,18 @@ class Eynollah: polygons_lines_xml = [] contours_tables = [] ocr_all_textlines = None + conf_contours_textregions =None pcgts = self.writer.build_pagexml_no_full_layout( cont_page, page_coord, order_text_new, id_of_texts_tot, all_found_textline_polygons, page_coord, polygons_of_images, polygons_of_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_marginals, - cont_page, polygons_lines_xml, contours_tables, ocr_all_textlines) + cont_page, polygons_lines_xml, contours_tables, ocr_all_textlines, conf_contours_textregions) return pcgts #print("text region early -1 in %.1fs", time.time() - t0) t1 = time.time() if self.light_version: - text_regions_p_1 ,erosion_hurts, polygons_lines_xml, textline_mask_tot_ea, img_bin_light = \ + text_regions_p_1 ,erosion_hurts, polygons_lines_xml, textline_mask_tot_ea, img_bin_light, confidence_matrix = \ self.get_regions_light_v(img_res, is_image_enhanced, num_col_classifier) #print("text region early -2 in %.1fs", time.time() - t0) @@ -4318,6 +4371,7 @@ class Eynollah: self.get_regions_from_xy_2models(img_res, is_image_enhanced, num_col_classifier) self.logger.info("Textregion detection took %.1fs ", time.time() - t1) + confidence_matrix = np.zeros((text_regions_p_1.shape[:2])) t1 = time.time() num_col, num_col_classifier, img_only_regions, page_coord, image_page, mask_images, mask_lines, \ @@ -4333,7 +4387,7 @@ class Eynollah: ocr_all_textlines = None pcgts = self.writer.build_pagexml_no_full_layout( [], page_coord, [], [], [], [], [], [], [], [], [], [], - cont_page, [], [], ocr_all_textlines) + cont_page, [], [], ocr_all_textlines, []) return pcgts #print("text region early in %.1fs", time.time() - t0) @@ -4512,29 +4566,35 @@ class Eynollah: [], [], page_coord, [], [], [], [], [], [], polygons_of_images, contours_tables, [], polygons_of_marginals, empty_marginals, empty_marginals, [], [], [], - cont_page, polygons_lines_xml, []) + cont_page, polygons_lines_xml, [], [], []) else: pcgts = self.writer.build_pagexml_no_full_layout( [], page_coord, [], [], [], [], polygons_of_images, polygons_of_marginals, empty_marginals, empty_marginals, [], [], - cont_page, polygons_lines_xml, contours_tables, []) + cont_page, polygons_lines_xml, contours_tables, [], []) return pcgts + + ## check the ro order + + + + #print("text region early 3 in %.1fs", time.time() - t0) if self.light_version: contours_only_text_parent = self.dilate_textregions_contours( contours_only_text_parent) - contours_only_text_parent = self.filter_contours_inside_a_bigger_one( - contours_only_text_parent, text_only, marginal_cnts=polygons_of_marginals) + contours_only_text_parent , contours_only_text_parent_d_ordered = self.filter_contours_inside_a_bigger_one( + contours_only_text_parent, contours_only_text_parent_d_ordered, text_only, marginal_cnts=polygons_of_marginals) #print("text region early 3.5 in %.1fs", time.time() - t0) - txt_con_org = get_textregion_contours_in_org_image_light( - contours_only_text_parent, self.image, slope_first, map=self.executor.map) + txt_con_org , conf_contours_textregions = get_textregion_contours_in_org_image_light( + contours_only_text_parent, self.image, slope_first, confidence_matrix, map=self.executor.map) #txt_con_org = self.dilate_textregions_contours(txt_con_org) #contours_only_text_parent = self.dilate_textregions_contours(contours_only_text_parent) else: - txt_con_org = get_textregion_contours_in_org_image( - contours_only_text_parent, self.image, slope_first) + txt_con_org , conf_contours_textregions = get_textregion_contours_in_org_image_light( + contours_only_text_parent, self.image, slope_first, confidence_matrix, map=self.executor.map) #print("text region early 4 in %.1fs", time.time() - t0) boxes_text, _ = get_text_region_boxes_by_given_contours(contours_only_text_parent) boxes_marginals, _ = get_text_region_boxes_by_given_contours(polygons_of_marginals) @@ -4563,12 +4623,12 @@ class Eynollah: all_found_textline_polygons = self.dilate_textregions_contours_textline_version( all_found_textline_polygons) all_found_textline_polygons = self.filter_contours_inside_a_bigger_one( - all_found_textline_polygons, textline_mask_tot_ea_org, type_contour="textline") + all_found_textline_polygons, None, textline_mask_tot_ea_org, type_contour="textline") all_found_textline_polygons_marginals = self.dilate_textregions_contours_textline_version( all_found_textline_polygons_marginals) - contours_only_text_parent, txt_con_org, all_found_textline_polygons, contours_only_text_parent_d_ordered, \ + contours_only_text_parent, txt_con_org, conf_contours_textregions, all_found_textline_polygons, contours_only_text_parent_d_ordered, \ index_by_text_par_con = self.filter_contours_without_textline_inside( - contours_only_text_parent, txt_con_org, all_found_textline_polygons, contours_only_text_parent_d_ordered) + contours_only_text_parent, txt_con_org, all_found_textline_polygons, contours_only_text_parent_d_ordered, conf_contours_textregions) else: textline_mask_tot_ea = cv2.erode(textline_mask_tot_ea, kernel=KERNEL, iterations=1) all_found_textline_polygons, boxes_text, txt_con_org, contours_only_text_parent, all_box_coord, \ @@ -4629,9 +4689,10 @@ class Eynollah: fun = check_any_text_region_in_model_one_is_main_or_header text_regions_p, contours_only_text_parent, contours_only_text_parent_h, all_box_coord, all_box_coord_h, \ all_found_textline_polygons, all_found_textline_polygons_h, slopes, slopes_h, \ - contours_only_text_parent_d_ordered, contours_only_text_parent_h_d_ordered = fun( + contours_only_text_parent_d_ordered, contours_only_text_parent_h_d_ordered, \ + conf_contours_textregions, conf_contours_textregions_h = fun( text_regions_p, regions_fully, contours_only_text_parent, - all_box_coord, all_found_textline_polygons, slopes, contours_only_text_parent_d_ordered) + all_box_coord, all_found_textline_polygons, slopes, contours_only_text_parent_d_ordered, conf_contours_textregions) if self.plotter: self.plotter.save_plot_of_layout(text_regions_p, image_page) @@ -4708,7 +4769,7 @@ class Eynollah: all_found_textline_polygons, all_found_textline_polygons_h, all_box_coord, all_box_coord_h, polygons_of_images, contours_tables, polygons_of_drop_capitals, polygons_of_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_h, slopes_marginals, - cont_page, polygons_lines_xml, ocr_all_textlines) + cont_page, polygons_lines_xml, ocr_all_textlines, conf_contours_textregions, conf_contours_textregions_h) return pcgts contours_only_text_parent_h = None @@ -4787,7 +4848,7 @@ class Eynollah: txt_con_org, page_coord, order_text_new, id_of_texts_tot, all_found_textline_polygons, all_box_coord, polygons_of_images, polygons_of_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_marginals, - cont_page, polygons_lines_xml, contours_tables, ocr_all_textlines) + cont_page, polygons_lines_xml, contours_tables, ocr_all_textlines, conf_contours_textregions) return pcgts @@ -4797,19 +4858,27 @@ class Eynollah_ocr: dir_models, dir_xmls=None, dir_in=None, + dir_in_bin=None, dir_out=None, + dir_out_image_text=None, tr_ocr=False, export_textline_images_and_text=False, do_not_mask_with_textline_contour=False, + draw_texts_on_image=False, + prediction_with_both_of_rgb_and_bin=False, logger=None, ): self.dir_in = dir_in + self.dir_in_bin = dir_in_bin self.dir_out = dir_out self.dir_xmls = dir_xmls self.dir_models = dir_models self.tr_ocr = tr_ocr self.export_textline_images_and_text = export_textline_images_and_text self.do_not_mask_with_textline_contour = do_not_mask_with_textline_contour + self.draw_texts_on_image = draw_texts_on_image + self.dir_out_image_text = dir_out_image_text + self.prediction_with_both_of_rgb_and_bin = prediction_with_both_of_rgb_and_bin if tr_ocr: self.processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-printed") self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") @@ -4818,7 +4887,7 @@ class Eynollah_ocr: self.model_ocr.to(self.device) else: - self.model_ocr_dir = dir_models + "/model_3_new_ocrcnn"#"/model_0_ocr_cnnrnn"#"/model_23_ocr_cnnrnn" + self.model_ocr_dir = dir_models + "/model_step_150000_ocr"#"/model_0_ocr_cnnrnn"#"/model_23_ocr_cnnrnn" model_ocr = load_model(self.model_ocr_dir , compile=False) self.prediction_model = tf.keras.models.Model( @@ -4927,26 +4996,53 @@ class Eynollah_ocr: return peaks_final else: return None + + # Function to fit text inside the given area + def fit_text_single_line(self, draw, text, font_path, max_width, max_height): + initial_font_size = 50 + font_size = initial_font_size + while font_size > 10: # Minimum font size + font = ImageFont.truetype(font_path, font_size) + text_bbox = draw.textbbox((0, 0), text, font=font) # Get text bounding box + text_width = text_bbox[2] - text_bbox[0] + text_height = text_bbox[3] - text_bbox[1] + + if text_width <= max_width and text_height <= max_height: + return font # Return the best-fitting font + + font_size -= 2 # Reduce font size and retry + + return ImageFont.truetype(font_path, 10) # Smallest font fallback - def return_textlines_split_if_needed(self, textline_image): + def return_textlines_split_if_needed(self, textline_image, textline_image_bin): split_point = self.return_start_and_end_of_common_text_of_textline_ocr_without_common_section(textline_image) if split_point: image1 = textline_image[:, :split_point,:]# image.crop((0, 0, width2, height)) image2 = textline_image[:, split_point:,:]#image.crop((width1, 0, width, height)) - return [image1, image2] + if self.prediction_with_both_of_rgb_and_bin: + image1_bin = textline_image_bin[:, :split_point,:]# image.crop((0, 0, width2, height)) + image2_bin = textline_image_bin[:, split_point:,:]#image.crop((width1, 0, width, height)) + return [image1, image2], [image1_bin, image2_bin] + else: + return [image1, image2], None else: - return None + return None, None def preprocess_and_resize_image_for_ocrcnn_model(self, img, image_height, image_width): ratio = image_height /float(img.shape[0]) w_ratio = int(ratio * img.shape[1]) + if w_ratio <= image_width: width_new = w_ratio else: width_new = image_width + + if width_new == 0: + width_new = img.shape[1] + img = resize_image(img, image_height, width_new) img_fin = np.ones((image_height, image_width, 3))*255 - img_fin[:,:width_new,:] = img[:,:,:] + img_fin[:,:+width_new,:] = img[:,:,:] img_fin = img_fin / 255. return img_fin @@ -5006,7 +5102,7 @@ class Eynollah_ocr: cropped_lines.append(img_crop) cropped_lines_meging_indexing.append(0) else: - splited_images = self.return_textlines_split_if_needed(img_crop) + splited_images, _ = self.return_textlines_split_if_needed(img_crop, None) #print(splited_images) if splited_images: cropped_lines.append(splited_images[0]) @@ -5097,6 +5193,16 @@ class Eynollah_ocr: dir_xml = os.path.join(self.dir_xmls, file_name+'.xml') out_file_ocr = os.path.join(self.dir_out, file_name+'.xml') img = cv2.imread(dir_img) + if self.prediction_with_both_of_rgb_and_bin: + cropped_lines_bin = [] + dir_img_bin = os.path.join(self.dir_in_bin, file_name+'.png') + img_bin = cv2.imread(dir_img_bin) + + if self.draw_texts_on_image: + out_image_with_text = os.path.join(self.dir_out_image_text, file_name+'.png') + image_text = Image.new("RGB", (img.shape[1], img.shape[0]), "white") + draw = ImageDraw.Draw(image_text) + total_bb_coordinates = [] tree1 = ET.parse(dir_xml, parser = ET.XMLParser(encoding="utf-8")) root1=tree1.getroot() @@ -5126,9 +5232,16 @@ class Eynollah_ocr: x,y,w,h = cv2.boundingRect(textline_coords) + if self.draw_texts_on_image: + total_bb_coordinates.append([x,y,w,h]) + h2w_ratio = h/float(w) img_poly_on_img = np.copy(img) + if self.prediction_with_both_of_rgb_and_bin: + img_poly_on_img_bin = np.copy(img_bin) + img_crop_bin = img_poly_on_img_bin[y:y+h, x:x+w, :] + mask_poly = np.zeros(img.shape) mask_poly = cv2.fillPoly(mask_poly, pts=[textline_coords], color=(1, 1, 1)) @@ -5136,14 +5249,22 @@ class Eynollah_ocr: img_crop = img_poly_on_img[y:y+h, x:x+w, :] if not self.do_not_mask_with_textline_contour: img_crop[mask_poly==0] = 255 + if self.prediction_with_both_of_rgb_and_bin: + img_crop_bin[mask_poly==0] = 255 if not self.export_textline_images_and_text: - if h2w_ratio > 0.05: + if h2w_ratio > 0.1: img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(img_crop, image_height, image_width) cropped_lines.append(img_fin) cropped_lines_meging_indexing.append(0) + if self.prediction_with_both_of_rgb_and_bin: + img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(img_crop_bin, image_height, image_width) + cropped_lines_bin.append(img_fin) else: - splited_images = self.return_textlines_split_if_needed(img_crop) + if self.prediction_with_both_of_rgb_and_bin: + splited_images, splited_images_bin = self.return_textlines_split_if_needed(img_crop, img_crop_bin) + else: + splited_images, splited_images_bin = self.return_textlines_split_if_needed(img_crop, None) if splited_images: img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(splited_images[0], image_height, image_width) cropped_lines.append(img_fin) @@ -5152,19 +5273,28 @@ class Eynollah_ocr: cropped_lines.append(img_fin) cropped_lines_meging_indexing.append(-1) + + if self.prediction_with_both_of_rgb_and_bin: + img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(splited_images_bin[0], image_height, image_width) + cropped_lines_bin.append(img_fin) + img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(splited_images_bin[1], image_height, image_width) + cropped_lines_bin.append(img_fin) + else: img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(img_crop, image_height, image_width) cropped_lines.append(img_fin) cropped_lines_meging_indexing.append(0) + + if self.prediction_with_both_of_rgb_and_bin: + img_fin = self.preprocess_and_resize_image_for_ocrcnn_model(img_crop_bin, image_height, image_width) + cropped_lines_bin.append(img_fin) if self.export_textline_images_and_text: if child_textlines.tag.endswith("TextEquiv"): for cheild_text in child_textlines: if cheild_text.tag.endswith("Unicode"): textline_text = cheild_text.text - if not textline_text: - pass - else: + if textline_text: with open(os.path.join(self.dir_out, file_name+'_line_'+str(indexer_textlines)+'.txt'), 'w') as text_file: text_file.write(textline_text) @@ -5186,24 +5316,65 @@ class Eynollah_ocr: imgs = cropped_lines[n_start:] imgs = np.array(imgs) imgs = imgs.reshape(imgs.shape[0], image_height, image_width, 3) + if self.prediction_with_both_of_rgb_and_bin: + imgs_bin = cropped_lines_bin[n_start:] + imgs_bin = np.array(imgs_bin) + imgs_bin = imgs_bin.reshape(imgs_bin.shape[0], image_height, image_width, 3) else: n_start = i*b_s n_end = (i+1)*b_s imgs = cropped_lines[n_start:n_end] imgs = np.array(imgs).reshape(b_s, image_height, image_width, 3) + if self.prediction_with_both_of_rgb_and_bin: + imgs_bin = cropped_lines_bin[n_start:n_end] + imgs_bin = np.array(imgs_bin).reshape(b_s, image_height, image_width, 3) + preds = self.prediction_model.predict(imgs, verbose=0) + if self.prediction_with_both_of_rgb_and_bin: + preds_bin = self.prediction_model.predict(imgs_bin, verbose=0) + preds = (preds + preds_bin) / 2. + pred_texts = self.decode_batch_predictions(preds) for ib in range(imgs.shape[0]): pred_texts_ib = pred_texts[ib].strip("[UNK]") extracted_texts.append(pred_texts_ib) - extracted_texts_merged = [extracted_texts[ind] if cropped_lines_meging_indexing[ind]==0 else extracted_texts[ind]+extracted_texts[ind+1] if cropped_lines_meging_indexing[ind]==1 else None for ind in range(len(cropped_lines_meging_indexing))] + extracted_texts_merged = [extracted_texts[ind] if cropped_lines_meging_indexing[ind]==0 else extracted_texts[ind]+" "+extracted_texts[ind+1] if cropped_lines_meging_indexing[ind]==1 else None for ind in range(len(cropped_lines_meging_indexing))] extracted_texts_merged = [ind for ind in extracted_texts_merged if ind is not None] unique_cropped_lines_region_indexer = np.unique(cropped_lines_region_indexer) + + + if self.draw_texts_on_image: + + font_path = "NotoSans-Regular.ttf" # Make sure this file exists! + font = ImageFont.truetype(font_path, 40) + + for indexer_text, bb_ind in enumerate(total_bb_coordinates): + + + x_bb = bb_ind[0] + y_bb = bb_ind[1] + w_bb = bb_ind[2] + h_bb = bb_ind[3] + + font = self.fit_text_single_line(draw, extracted_texts_merged[indexer_text], font_path, w_bb, int(h_bb*0.4) ) + + ##draw.rectangle([x_bb, y_bb, x_bb + w_bb, y_bb + h_bb], outline="red", width=2) + + text_bbox = draw.textbbox((0, 0), extracted_texts_merged[indexer_text], font=font) + text_width = text_bbox[2] - text_bbox[0] + text_height = text_bbox[3] - text_bbox[1] + + text_x = x_bb + (w_bb - text_width) // 2 # Center horizontally + text_y = y_bb + (h_bb - text_height) // 2 # Center vertically + + # Draw the text + draw.text((text_x, text_y), extracted_texts_merged[indexer_text], fill="black", font=font) + image_text.save(out_image_with_text) text_by_textregion = [] for ind in unique_cropped_lines_region_indexer: @@ -5213,20 +5384,49 @@ class Eynollah_ocr: indexer = 0 indexer_textregion = 0 for nn in root1.iter(region_tags): - text_subelement_textregion = ET.SubElement(nn, 'TextEquiv') - unicode_textregion = ET.SubElement(text_subelement_textregion, 'Unicode') + + is_textregion_text = False + for childtest in nn: + if childtest.tag.endswith("TextEquiv"): + is_textregion_text = True + + if not is_textregion_text: + text_subelement_textregion = ET.SubElement(nn, 'TextEquiv') + unicode_textregion = ET.SubElement(text_subelement_textregion, 'Unicode') has_textline = False for child_textregion in nn: if child_textregion.tag.endswith("TextLine"): - text_subelement = ET.SubElement(child_textregion, 'TextEquiv') - unicode_textline = ET.SubElement(text_subelement, 'Unicode') - unicode_textline.text = extracted_texts_merged[indexer] + + is_textline_text = False + for childtest2 in child_textregion: + if childtest2.tag.endswith("TextEquiv"): + is_textline_text = True + + + if not is_textline_text: + text_subelement = ET.SubElement(child_textregion, 'TextEquiv') + unicode_textline = ET.SubElement(text_subelement, 'Unicode') + unicode_textline.text = extracted_texts_merged[indexer] + else: + for childtest3 in child_textregion: + if childtest3.tag.endswith("TextEquiv"): + for child_uc in childtest3: + if child_uc.tag.endswith("Unicode"): + child_uc.text = extracted_texts_merged[indexer] + indexer = indexer + 1 has_textline = True if has_textline: - unicode_textregion.text = text_by_textregion[indexer_textregion] + if is_textregion_text: + for child4 in nn: + if child4.tag.endswith("TextEquiv"): + for childtr_uc in child4: + if childtr_uc.tag.endswith("Unicode"): + childtr_uc.text = text_by_textregion[indexer_textregion] + else: + unicode_textregion.text = text_by_textregion[indexer_textregion] indexer_textregion = indexer_textregion + 1 ET.register_namespace("",name_space) diff --git a/src/eynollah/utils/__init__.py b/src/eynollah/utils/__init__.py index b93801c..c5962f8 100644 --- a/src/eynollah/utils/__init__.py +++ b/src/eynollah/utils/__init__.py @@ -8,7 +8,6 @@ except ImportError: import numpy as np from shapely import geometry import cv2 -import imutils from scipy.signal import find_peaks from scipy.ndimage import gaussian_filter1d @@ -28,7 +27,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( y_sep=[] y_diff=[] new_main_sep_y=[] - + indexer=0 for i in range(len(x_min_hor_some)): starting=x_min_hor_some[i]-peak_points @@ -36,34 +35,34 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( min_start=np.argmin(starting) ending=peak_points-x_max_hor_some[i] len_ending_neg=len(ending[ending<=0]) - + ending=ending[ending>0] max_end=np.argmin(ending)+len_ending_neg - + if (max_end-min_start)>=2: if (max_end-min_start)==(len(peak_points)-1): new_main_sep_y.append(indexer) - + #print((max_end-min_start),len(peak_points),'(max_end-min_start)') y_sep.append(cy_hor_some[i]) y_diff.append(cy_hor_diff[i]) x_end.append(max_end) - + x_start.append( min_start) - + len_sep.append(max_end-min_start) if max_end==min_start+1: kind.append(0) else: kind.append(1) - + indexer+=1 x_start_returned = np.array(x_start, dtype=int) x_end_returned = np.array(x_end, dtype=int) y_sep_returned = np.array(y_sep, dtype=int) y_diff_returned = np.array(y_diff, dtype=int) - + all_args_uniq = contours_in_same_horizon(y_sep_returned) args_to_be_unified=[] y_unified=[] @@ -92,7 +91,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( y_diff_selected=np.max(y_diff_same_hor) x_s_selected=np.min(x_s_same_hor) x_e_selected=np.max(x_e_same_hor) - + x_s_unified.append(x_s_selected) x_e_unified.append(x_e_selected) y_unified.append(y_selected) @@ -106,56 +105,56 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( args_lines_not_unified=list( set(range(len(y_sep_returned)))-set(args_to_be_unified) ) #print(args_lines_not_unified,'args_lines_not_unified') - + x_start_returned_not_unified=list( np.array(x_start_returned)[args_lines_not_unified] ) x_end_returned_not_unified=list( np.array(x_end_returned)[args_lines_not_unified] ) y_sep_returned_not_unified=list (np.array(y_sep_returned)[args_lines_not_unified] ) y_diff_returned_not_unified=list (np.array(y_diff_returned)[args_lines_not_unified] ) - + for dv in range(len(y_unified)): y_sep_returned_not_unified.append(y_unified[dv]) y_diff_returned_not_unified.append(y_diff_unified[dv]) x_start_returned_not_unified.append(x_s_unified[dv]) x_end_returned_not_unified.append(x_e_unified[dv]) - + #print(y_sep_returned,'y_sep_returned') #print(x_start_returned,'x_start_returned') #print(x_end_returned,'x_end_returned') - + x_start_returned = np.array(x_start_returned_not_unified, dtype=int) x_end_returned = np.array(x_end_returned_not_unified, dtype=int) y_sep_returned = np.array(y_sep_returned_not_unified, dtype=int) y_diff_returned = np.array(y_diff_returned_not_unified, dtype=int) - + #print(y_sep_returned,'y_sep_returned2') #print(x_start_returned,'x_start_returned2') #print(x_end_returned,'x_end_returned2') #print(new_main_sep_y,'new_main_sep_y') - + #print(x_start,'x_start') #print(x_end,'x_end') if len(new_main_sep_y)>0: - + min_ys=np.min(y_sep) max_ys=np.max(y_sep) - + y_mains=[] y_mains.append(min_ys) y_mains_sep_ohne_grenzen=[] - + for ii in range(len(new_main_sep_y)): y_mains.append(y_sep[new_main_sep_y[ii]]) y_mains_sep_ohne_grenzen.append(y_sep[new_main_sep_y[ii]]) - + y_mains.append(max_ys) - + y_mains_sorted=np.sort(y_mains) diff=np.diff(y_mains_sorted) argm=np.argmax(diff) - + y_min_new=y_mains_sorted[argm] y_max_new=y_mains_sorted[argm+1] - + #print(y_min_new,'y_min_new') #print(y_max_new,'y_max_new') #print(y_sep[new_main_sep_y[0]],y_sep,'yseps') @@ -192,7 +191,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( #print(x_start,'x_start') #print(x_end,'x_end') #print(len_sep) - + deleted=[] for i in range(len(x_start)-1): nodes_i=set(range(x_start[i],x_end[i]+1)) @@ -200,10 +199,10 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( if nodes_i==set(range(x_start[j],x_end[j]+1)): deleted.append(j) #print(np.unique(deleted)) - + remained_sep_indexes=set(range(len(x_start)))-set(np.unique(deleted) ) #print(remained_sep_indexes,'remained_sep_indexes') - mother=[]#if it has mother + mother=[]#if it has mother child=[] for index_i in remained_sep_indexes: have_mother=0 @@ -217,14 +216,14 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( have_child=1 mother.append(have_mother) child.append(have_child) - + #print(mother,'mother') #print(len(remained_sep_indexes)) #print(len(remained_sep_indexes),len(x_start),len(x_end),len(y_sep),'lens') y_lines_without_mother=[] x_start_without_mother=[] x_end_without_mother=[] - + y_lines_with_child_without_mother=[] x_start_with_child_without_mother=[] x_end_with_child_without_mother=[] @@ -237,7 +236,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( x_start = np.array(x_start) x_end = np.array(x_end) y_sep = np.array(y_sep) - + if len(remained_sep_indexes)>1: #print(np.array(remained_sep_indexes),'np.array(remained_sep_indexes)') #print(np.array(mother),'mother') @@ -245,7 +244,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( remained_sep_indexes_with_child_without_mother = remained_sep_indexes[(mother==0) & (child==1)] #print(remained_sep_indexes_without_mother,'remained_sep_indexes_without_mother') #print(remained_sep_indexes_without_mother,'remained_sep_indexes_without_mother') - + x_end_with_child_without_mother = x_end[remained_sep_indexes_with_child_without_mother] x_start_with_child_without_mother = x_start[remained_sep_indexes_with_child_without_mother] y_lines_with_child_without_mother = y_sep[remained_sep_indexes_with_child_without_mother] @@ -254,7 +253,7 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( x_end_without_mother = x_end[remained_sep_indexes_without_mother] x_start_without_mother = x_start[remained_sep_indexes_without_mother] y_lines_without_mother = y_sep[remained_sep_indexes_without_mother] - + if len(remained_sep_indexes_without_mother)>=2: for i in range(len(remained_sep_indexes_without_mother)-1): nodes_i=set(range(x_start[remained_sep_indexes_without_mother[i]], @@ -275,16 +274,16 @@ def return_x_start_end_mothers_childs_and_type_of_reading_order( #print(y_lines_with_child_without_mother,'y_lines_with_child_without_mother') #print(x_start_with_child_without_mother,'x_start_with_child_without_mother') #print(x_end_with_child_without_mother,'x_end_with_hild_without_mother') - + len_sep_with_child = len(child[child==1]) - + #print(len_sep_with_child,'len_sep_with_child') there_is_sep_with_child = 0 if len_sep_with_child >= 1: there_is_sep_with_child = 1 #print(all_args_uniq,'all_args_uniq') #print(args_to_be_unified,'args_to_be_unified') - + return (reading_orther_type, x_start_returned, x_end_returned, @@ -433,13 +432,13 @@ def find_num_col(regions_without_separators, num_col_classifier, tables, multipl interest_neg_fin = interest_neg[(interest_neg < grenze)] peaks_neg_fin = peaks_neg[(interest_neg < grenze)] # interest_neg_fin=interest_neg[(interest_neg= 3: index_sort_interest_neg_fin= np.argsort(interest_neg_fin) peaks_neg_sorted = np.array(peaks_neg)[index_sort_interest_neg_fin] interest_neg_fin_sorted = np.array(interest_neg_fin)[index_sort_interest_neg_fin] - + if len(index_sort_interest_neg_fin)>=num_col_classifier: peaks_neg_fin = list( peaks_neg_sorted[:num_col_classifier] ) interest_neg_fin = list( interest_neg_fin_sorted[:num_col_classifier] ) @@ -846,11 +845,11 @@ def putt_bb_of_drop_capitals_of_model_in_patches_in_layout(layout_in_patch, drop box0 = box + (0,) mask_of_drop_cpaital_in_early_layout = np.zeros((text_regions_p.shape[0], text_regions_p.shape[1])) mask_of_drop_cpaital_in_early_layout[box] = text_regions_p[box] - + all_drop_capital_pixels_which_is_text_in_early_lo = np.sum(mask_of_drop_cpaital_in_early_layout[box]==1) mask_of_drop_cpaital_in_early_layout[box] = 1 all_drop_capital_pixels = np.sum(mask_of_drop_cpaital_in_early_layout==1) - + percent_text_to_all_in_drop = all_drop_capital_pixels_which_is_text_in_early_lo / float(all_drop_capital_pixels) if (areas_cnt_text[jj] * float(drop_only.shape[0] * drop_only.shape[1]) / float(w * h) > 0.6 and percent_text_to_all_in_drop >= 0.3): @@ -868,7 +867,7 @@ def check_any_text_region_in_model_one_is_main_or_header( contours_only_text_parent, all_box_coord, all_found_textline_polygons, slopes, - contours_only_text_parent_d_ordered): + contours_only_text_parent_d_ordered, conf_contours): cx_main, cy_main, x_min_main, x_max_main, y_min_main, y_max_main, y_corr_x_min_from_argmin = \ find_new_features_of_contours(contours_only_text_parent) @@ -888,6 +887,9 @@ def check_any_text_region_in_model_one_is_main_or_header( contours_only_text_parent_main=[] contours_only_text_parent_head=[] + conf_contours_main=[] + conf_contours_head=[] + contours_only_text_parent_main_d=[] contours_only_text_parent_head_d=[] @@ -908,9 +910,11 @@ def check_any_text_region_in_model_one_is_main_or_header( all_box_coord_head.append(all_box_coord[ii]) slopes_head.append(slopes[ii]) all_found_textline_polygons_head.append(all_found_textline_polygons[ii]) + conf_contours_head.append(None) else: regions_model_1[:,:][(regions_model_1[:,:]==1) & (img[:,:,0]==255) ]=1 contours_only_text_parent_main.append(con) + conf_contours_main.append(conf_contours[ii]) if contours_only_text_parent_d_ordered is not None: contours_only_text_parent_main_d.append(contours_only_text_parent_d_ordered[ii]) all_box_coord_main.append(all_box_coord[ii]) @@ -929,14 +933,17 @@ def check_any_text_region_in_model_one_is_main_or_header( slopes_main, slopes_head, contours_only_text_parent_main_d, - contours_only_text_parent_head_d) + contours_only_text_parent_head_d, + conf_contours_main, + conf_contours_head) def check_any_text_region_in_model_one_is_main_or_header_light( regions_model_1, regions_model_full, contours_only_text_parent, all_box_coord, all_found_textline_polygons, slopes, - contours_only_text_parent_d_ordered): + contours_only_text_parent_d_ordered, + conf_contours): ### to make it faster h_o = regions_model_1.shape[0] @@ -969,6 +976,9 @@ def check_any_text_region_in_model_one_is_main_or_header_light( contours_only_text_parent_main=[] contours_only_text_parent_head=[] + conf_contours_main=[] + conf_contours_head=[] + contours_only_text_parent_main_d=[] contours_only_text_parent_head_d=[] @@ -990,9 +1000,11 @@ def check_any_text_region_in_model_one_is_main_or_header_light( all_box_coord_head.append(all_box_coord[ii]) slopes_head.append(slopes[ii]) all_found_textline_polygons_head.append(all_found_textline_polygons[ii]) + conf_contours_head.append(None) else: regions_model_1[:,:][(regions_model_1[:,:]==1) & (img[:,:,0]==255) ]=1 contours_only_text_parent_main.append(con) + conf_contours_main.append(conf_contours[ii]) if contours_only_text_parent_d_ordered is not None: contours_only_text_parent_main_d.append(contours_only_text_parent_d_ordered[ii]) all_box_coord_main.append(all_box_coord[ii]) @@ -1009,7 +1021,7 @@ def check_any_text_region_in_model_one_is_main_or_header_light( contours_only_text_parent_head = [(i * zoom).astype(int) for i in contours_only_text_parent_head] contours_only_text_parent_main = [(i * zoom).astype(int) for i in contours_only_text_parent_main] ### - + return (regions_model_1, contours_only_text_parent_main, contours_only_text_parent_head, @@ -1020,7 +1032,9 @@ def check_any_text_region_in_model_one_is_main_or_header_light( slopes_main, slopes_head, contours_only_text_parent_main_d, - contours_only_text_parent_head_d) + contours_only_text_parent_head_d, + conf_contours_main, + conf_contours_head) def small_textlines_to_parent_adherence2(textlines_con, textline_iamge, num_col): # print(textlines_con) @@ -1317,11 +1331,11 @@ def combine_hor_lines_and_delete_cross_points_and_get_lines_features_back_new( imgray = cv2.cvtColor(img_in_hor, cv2.COLOR_BGR2GRAY) ret, thresh = cv2.threshold(imgray, 0, 255, 0) contours_lines_hor,hierarchy=cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) - + slope_lines_hor, dist_x_hor, x_min_main_hor, x_max_main_hor, cy_main_hor, _, _, _, _ = \ find_features_of_lines(contours_lines_hor) x_width_smaller_than_acolumn_width=img_in_hor.shape[1]/float(num_col_classifier+1.) - + len_lines_bigger_than_x_width_smaller_than_acolumn_width=len( dist_x_hor[dist_x_hor>=x_width_smaller_than_acolumn_width] ) len_lines_bigger_than_x_width_smaller_than_acolumn_width_per_column=int(len_lines_bigger_than_x_width_smaller_than_acolumn_width / float(num_col_classifier)) @@ -1339,7 +1353,7 @@ def combine_hor_lines_and_delete_cross_points_and_get_lines_features_back_new( some_cy=cy_main_hor[all_args_uniq[dd]] some_x_min=x_min_main_hor[all_args_uniq[dd]] some_x_max=x_max_main_hor[all_args_uniq[dd]] - + #img_in=np.zeros(separators_closeup_n[:,:,2].shape) #print(img_p_in_ver.shape[1],some_x_max-some_x_min,'xdiff') diff_x_some=some_x_max-some_x_min @@ -1352,7 +1366,7 @@ def combine_hor_lines_and_delete_cross_points_and_get_lines_features_back_new( int(np.max(some_x_max)) ]=1 sum_dis=dist_x_hor[some_args].sum() diff_max_min_uniques=np.max(x_max_main_hor[some_args])-np.min(x_min_main_hor[some_args]) - + if (diff_max_min_uniques > sum_dis and sum_dis / float(diff_max_min_uniques) > 0.85 and diff_max_min_uniques / float(img_p_in_ver.shape[1]) > 0.85 and @@ -1371,7 +1385,7 @@ def combine_hor_lines_and_delete_cross_points_and_get_lines_features_back_new( else: img_p_in=img_in_hor special_separators=[] - + img_p_in_ver[:,:,0][img_p_in_ver[:,:,0]==255]=1 sep_ver_hor=img_p_in+img_p_in_ver sep_ver_hor_cross=(sep_ver_hor[:,:,0]==2)*1 @@ -1402,7 +1416,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, separators_closeup=( (region_pre_p[:,:,:]==pixel_lines))*1 separators_closeup[0:110,:,:]=0 separators_closeup[separators_closeup.shape[0]-150:,:,:]=0 - + kernel = np.ones((5,5),np.uint8) separators_closeup=separators_closeup.astype(np.uint8) separators_closeup = cv2.dilate(separators_closeup,kernel,iterations = 1) @@ -1420,7 +1434,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, gray_early=gray_early.astype(np.uint8) imgray_e = cv2.cvtColor(gray_early, cv2.COLOR_BGR2GRAY) ret_e, thresh_e = cv2.threshold(imgray_e, 0, 255, 0) - + contours_line_e,hierarchy_e=cv2.findContours(thresh_e,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) _, dist_xe, _, _, _, _, y_min_main, y_max_main, _ = \ find_features_of_lines(contours_line_e) @@ -1433,11 +1447,11 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, cnts_hor_e.append(contours_line_e[ce]) figs_e=np.zeros(thresh_e.shape) figs_e=cv2.fillPoly(figs_e,pts=cnts_hor_e,color=(1,1,1)) - + separators_closeup_n_binary=cv2.fillPoly(separators_closeup_n_binary, pts=cnts_hor_e, color=(0,0,0)) gray = cv2.bitwise_not(separators_closeup_n_binary) gray=gray.astype(np.uint8) - + bw = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, \ cv2.THRESH_BINARY, 15, -2) horizontal = np.copy(bw) @@ -1455,7 +1469,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, horizontal = cv2.dilate(horizontal,kernel,iterations = 2) horizontal = cv2.erode(horizontal,kernel,iterations = 2) horizontal = cv2.fillPoly(horizontal, pts=cnts_hor_e, color=(255,255,255)) - + rows = vertical.shape[0] verticalsize = rows // 30 # Create structure element for extracting vertical lines through morphology operations @@ -1468,16 +1482,16 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, horizontal, special_separators = \ combine_hor_lines_and_delete_cross_points_and_get_lines_features_back_new( vertical, horizontal, num_col_classifier) - + separators_closeup_new[:,:][vertical[:,:]!=0]=1 separators_closeup_new[:,:][horizontal[:,:]!=0]=1 - + vertical=np.repeat(vertical[:, :, np.newaxis], 3, axis=2) vertical=vertical.astype(np.uint8) - + imgray = cv2.cvtColor(vertical, cv2.COLOR_BGR2GRAY) ret, thresh = cv2.threshold(imgray, 0, 255, 0) - + contours_line_vers,hierarchy=cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) slope_lines, dist_x, x_min_main, x_max_main, cy_main, slope_lines_org, y_min_main, y_max_main, cx_main = \ find_features_of_lines(contours_line_vers) @@ -1492,7 +1506,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, cx_main_ver=cx_main[slope_lines==1] dist_y_ver=y_max_main_ver-y_min_main_ver len_y=separators_closeup.shape[0]/3.0 - + horizontal=np.repeat(horizontal[:, :, np.newaxis], 3, axis=2) horizontal=horizontal.astype(np.uint8) imgray = cv2.cvtColor(horizontal, cv2.COLOR_BGR2GRAY) @@ -1500,12 +1514,12 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, contours_line_hors,hierarchy=cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) slope_lines, dist_x, x_min_main, x_max_main, cy_main, slope_lines_org, y_min_main, y_max_main, cx_main = \ find_features_of_lines(contours_line_hors) - + slope_lines_org_hor=slope_lines_org[slope_lines==0] args=np.arange(len(slope_lines)) len_x=separators_closeup.shape[1]/5.0 dist_y=np.abs(y_max_main-y_min_main) - + args_hor=args[slope_lines==0] dist_x_hor=dist_x[slope_lines==0] y_min_main_hor=y_min_main[slope_lines==0] @@ -1524,7 +1538,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, dist_y_hor=dist_y_hor[dist_x_hor>=len_x/2.0] slope_lines_org_hor=slope_lines_org_hor[dist_x_hor>=len_x/2.0] dist_x_hor=dist_x_hor[dist_x_hor>=len_x/2.0] - + matrix_of_lines_ch=np.zeros((len(cy_main_hor)+len(cx_main_ver),10)) matrix_of_lines_ch[:len(cy_main_hor),0]=args_hor matrix_of_lines_ch[len(cy_main_hor):,0]=args_ver @@ -1543,14 +1557,14 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, matrix_of_lines_ch[:len(cy_main_hor),8]=dist_y_hor matrix_of_lines_ch[len(cy_main_hor):,8]=dist_y_ver matrix_of_lines_ch[len(cy_main_hor):,9]=1 - + if contours_h is not None: _, dist_x_head, x_min_main_head, x_max_main_head, cy_main_head, _, y_min_main_head, y_max_main_head, _ = \ find_features_of_lines(contours_h) matrix_l_n=np.zeros((matrix_of_lines_ch.shape[0]+len(cy_main_head),matrix_of_lines_ch.shape[1])) matrix_l_n[:matrix_of_lines_ch.shape[0],:]=np.copy(matrix_of_lines_ch[:,:]) args_head=np.arange(len(cy_main_head)) + len(cy_main_hor) - + matrix_l_n[matrix_of_lines_ch.shape[0]:,0]=args_head matrix_l_n[matrix_of_lines_ch.shape[0]:,2]=x_min_main_head+30 matrix_l_n[matrix_of_lines_ch.shape[0]:,3]=x_max_main_head-30 @@ -1560,7 +1574,7 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, matrix_l_n[matrix_of_lines_ch.shape[0]:,7]=y_max_main_head#y_min_main_head+1-8 matrix_l_n[matrix_of_lines_ch.shape[0]:,8]=4 matrix_of_lines_ch=np.copy(matrix_l_n) - + cy_main_splitters=cy_main_hor[(x_min_main_hor<=.16*region_pre_p.shape[1]) & (x_max_main_hor>=.84*region_pre_p.shape[1])] cy_main_splitters=np.array( list(cy_main_splitters)+list(special_separators)) @@ -1573,19 +1587,19 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, pass args_cy_splitter=np.argsort(cy_main_splitters) cy_main_splitters_sort=cy_main_splitters[args_cy_splitter] - + splitter_y_new=[] splitter_y_new.append(0) for i in range(len(cy_main_splitters_sort)): - splitter_y_new.append( cy_main_splitters_sort[i] ) + splitter_y_new.append( cy_main_splitters_sort[i] ) splitter_y_new.append(region_pre_p.shape[0]) splitter_y_new_diff=np.diff(splitter_y_new)/float(region_pre_p.shape[0])*100 - + args_big_parts=np.arange(len(splitter_y_new_diff))[ splitter_y_new_diff>22 ] regions_without_separators=return_regions_without_separators(region_pre_p) length_y_threshold=regions_without_separators.shape[0]/4.0 - + num_col_fin=0 peaks_neg_fin_fin=[] for itiles in args_big_parts: @@ -1600,15 +1614,15 @@ def find_number_of_columns_in_document(region_pre_p, num_col_classifier, tables, if num_col>num_col_fin: num_col_fin=num_col peaks_neg_fin_fin=peaks_neg_fin - + if len(args_big_parts)==1 and (len(peaks_neg_fin_fin)+1)=500] peaks_neg_fin=peaks_neg_fin[peaks_neg_fin<=(vertical.shape[1]-500)] peaks_neg_fin_fin=peaks_neg_fin[:] - + return num_col_fin, peaks_neg_fin_fin,matrix_of_lines_ch,splitter_y_new,separators_closeup_n - + def return_boxes_of_images_by_order_of_reading_new( splitter_y_new, regions_without_separators, matrix_of_lines_ch, @@ -1655,7 +1669,7 @@ def return_boxes_of_images_by_order_of_reading_new( for p_n in peaks_neg_fin: peaks_neg_fin_early.append(p_n) peaks_neg_fin_early.append(regions_without_separators.shape[1]-1) - + #print(peaks_neg_fin_early,'burda2') peaks_neg_fin_rev=[] for i_n in range(len(peaks_neg_fin_early)-1): @@ -1679,25 +1693,25 @@ def return_boxes_of_images_by_order_of_reading_new( num_col_classifier,tables, multiplier=5.) except: peaks_neg_fin2=[] - + if len(peaks_neg_fin1)>=len(peaks_neg_fin2): peaks_neg_fin=list(np.copy(peaks_neg_fin1)) else: peaks_neg_fin=list(np.copy(peaks_neg_fin2)) peaks_neg_fin=list(np.array(peaks_neg_fin)+peaks_neg_fin_early[i_n]) - + if i_n!=(len(peaks_neg_fin_early)-2): peaks_neg_fin_rev.append(peaks_neg_fin_early[i_n+1]) #print(peaks_neg_fin,'peaks_neg_fin') peaks_neg_fin_rev=peaks_neg_fin_rev+peaks_neg_fin - if len(peaks_neg_fin_rev)>=len(peaks_neg_fin_org): + if len(peaks_neg_fin_rev)>=len(peaks_neg_fin_org): peaks_neg_fin=list(np.sort(peaks_neg_fin_rev)) num_col=len(peaks_neg_fin) else: peaks_neg_fin=list(np.copy(peaks_neg_fin_org)) num_col=len(peaks_neg_fin) - + #print(peaks_neg_fin,'peaks_neg_fin') except: pass @@ -1709,16 +1723,16 @@ def return_boxes_of_images_by_order_of_reading_new( cy_hor_some=matrix_new[:,5][ (matrix_new[:,9]==0) ] cy_hor_diff=matrix_new[:,7][ (matrix_new[:,9]==0) ] arg_org_hor_some=matrix_new[:,0][ (matrix_new[:,9]==0) ] - + if right2left_readingorder: x_max_hor_some_new = regions_without_separators.shape[1] - x_min_hor_some x_min_hor_some_new = regions_without_separators.shape[1] - x_max_hor_some x_min_hor_some =list(np.copy(x_min_hor_some_new)) x_max_hor_some =list(np.copy(x_max_hor_some_new)) - + peaks_neg_tot=return_points_with_boundies(peaks_neg_fin,0, regions_without_separators[:,:].shape[1]) peaks_neg_tot_tables.append(peaks_neg_tot) - + reading_order_type, x_starting, x_ending, y_type_2, y_diff_type_2, \ y_lines_without_mother, x_start_without_mother, x_end_without_mother, there_is_sep_with_child, \ y_lines_with_child_without_mother, x_start_with_child_without_mother, x_end_with_child_without_mother, \ @@ -1735,7 +1749,7 @@ def return_boxes_of_images_by_order_of_reading_new( try: y_grenze=int(splitter_y_new[i])+300 #check if there is a big separator in this y_mains_sep_ohne_grenzen - + args_early_ys=np.arange(len(y_type_2)) #print(args_early_ys,'args_early_ys') #print(int(splitter_y_new[i]),int(splitter_y_new[i+1])) @@ -1764,13 +1778,13 @@ def return_boxes_of_images_by_order_of_reading_new( boxes.append([0, peaks_neg_tot[len(peaks_neg_tot)-1], int(splitter_y_new[i]), int( np.max(y_diff_main_separator_up))]) splitter_y_new[i]=[ np.max(y_diff_main_separator_up) ][0] - + #print(splitter_y_new[i],'splitter_y_new[i]') y_type_2 = y_type_2[args_to_be_kept] x_starting = x_starting[args_to_be_kept] x_ending = x_ending[args_to_be_kept] y_diff_type_2 = y_diff_type_2[args_to_be_kept] - + #print('galdiha') y_grenze=int(splitter_y_new[i])+200 args_early_ys2=np.arange(len(y_type_2)) @@ -1791,7 +1805,7 @@ def return_boxes_of_images_by_order_of_reading_new( x_ending_up[ij])) nodes_in = np.unique(nodes_in) #print(nodes_in,'nodes_in') - + if set(nodes_in)==set(range(len(peaks_neg_tot)-1)): pass elif set(nodes_in)==set(range(1, len(peaks_neg_tot)-1)): @@ -1799,7 +1813,7 @@ def return_boxes_of_images_by_order_of_reading_new( else: #print('burdaydikh') args_to_be_kept2=np.array(list( set(args_early_ys2)-set(args_up2) )) - + if len(args_to_be_kept2)>0: y_type_2 = y_type_2[args_to_be_kept2] x_starting = x_starting[args_to_be_kept2] @@ -1816,7 +1830,7 @@ def return_boxes_of_images_by_order_of_reading_new( nodes_in = np.unique(nodes_in) #print(nodes_in,'nodes_in2') #print(np.array(range(len(peaks_neg_tot)-1)),'np.array(range(len(peaks_neg_tot)-1))') - + if set(nodes_in)==set(range(len(peaks_neg_tot)-1)): pass elif set(nodes_in)==set(range(1,len(peaks_neg_tot)-1)): @@ -1826,7 +1840,7 @@ def return_boxes_of_images_by_order_of_reading_new( #print(args_early_ys,'args_early_ys') #print(args_up,'args_up') args_to_be_kept2=np.array(list( set(args_early_ys) - set(args_up) )) - + #print(args_to_be_kept2,'args_to_be_kept2') #print(len(y_type_2),len(x_starting),len(x_ending),len(y_diff_type_2)) if len(args_to_be_kept2)>0: @@ -1837,7 +1851,7 @@ def return_boxes_of_images_by_order_of_reading_new( else: pass #print('burdaydikh2') - + #int(splitter_y_new[i]) y_lines_by_order=[] x_start_by_order=[] @@ -1898,7 +1912,7 @@ def return_boxes_of_images_by_order_of_reading_new( list(range(x_start_without_mother[dj], x_end_without_mother[dj])) columns_covered_by_mothers = list(set(columns_covered_by_mothers)) - + all_columns=np.arange(len(peaks_neg_tot)-1) columns_not_covered=list(set(all_columns) - set(columns_covered_by_mothers)) y_type_2 = np.append(y_type_2, [int(splitter_y_new[i])] * (len(columns_not_covered) + len(x_start_without_mother))) @@ -1908,14 +1922,14 @@ def return_boxes_of_images_by_order_of_reading_new( x_starting = np.append(x_starting, x_start_without_mother) x_ending = np.append(x_ending, np.array(columns_not_covered) + 1) x_ending = np.append(x_ending, x_end_without_mother) - + columns_covered_by_with_child_no_mothers = [] for dj in range(len(x_end_with_child_without_mother)): columns_covered_by_with_child_no_mothers = columns_covered_by_with_child_no_mothers + \ list(range(x_start_with_child_without_mother[dj], x_end_with_child_without_mother[dj])) columns_covered_by_with_child_no_mothers = list(set(columns_covered_by_with_child_no_mothers)) - + all_columns = np.arange(len(peaks_neg_tot)-1) columns_not_covered_child_no_mother = list(set(all_columns) - set(columns_covered_by_with_child_no_mothers)) #indexes_to_be_spanned=[] @@ -1952,17 +1966,17 @@ def return_boxes_of_images_by_order_of_reading_new( x_diff_all_between_nm_wc = x_ending_all_between_nm_wc - x_starting_all_between_nm_wc if len(x_diff_all_between_nm_wc)>0: biggest=np.argmax(x_diff_all_between_nm_wc) - + columns_covered_by_mothers = [] for dj in range(len(x_starting_all_between_nm_wc)): columns_covered_by_mothers = columns_covered_by_mothers + \ list(range(x_starting_all_between_nm_wc[dj], x_ending_all_between_nm_wc[dj])) columns_covered_by_mothers = list(set(columns_covered_by_mothers)) - + all_columns=np.arange(i_s_nc, x_end_biggest_column) columns_not_covered = list(set(all_columns) - set(columns_covered_by_mothers)) - + should_longest_line_be_extended=0 if (len(x_diff_all_between_nm_wc) > 0 and set(list(range(x_starting_all_between_nm_wc[biggest], @@ -1980,11 +1994,11 @@ def return_boxes_of_images_by_order_of_reading_new( y_all_between_nm_wc = y_all_between_nm_wc[indexes_remained_after_deleting_closed_lines] x_starting_all_between_nm_wc = x_starting_all_between_nm_wc[indexes_remained_after_deleting_closed_lines] x_ending_all_between_nm_wc = x_ending_all_between_nm_wc[indexes_remained_after_deleting_closed_lines] - + y_all_between_nm_wc = np.append(y_all_between_nm_wc, y_column_nc[i_c]) x_starting_all_between_nm_wc = np.append(x_starting_all_between_nm_wc, i_s_nc) x_ending_all_between_nm_wc = np.append(x_ending_all_between_nm_wc, x_end_biggest_column) - + if len(x_diff_all_between_nm_wc) > 0: try: y_all_between_nm_wc = np.append(y_all_between_nm_wc, y_column_nc[i_c]) @@ -1992,11 +2006,11 @@ def return_boxes_of_images_by_order_of_reading_new( x_ending_all_between_nm_wc = np.append(x_ending_all_between_nm_wc, x_ending_all_between_nm_wc[biggest]) except: pass - + y_all_between_nm_wc = np.append(y_all_between_nm_wc, [y_column_nc[i_c]] * len(columns_not_covered)) x_starting_all_between_nm_wc = np.append(x_starting_all_between_nm_wc, columns_not_covered) x_ending_all_between_nm_wc = np.append(x_ending_all_between_nm_wc, np.array(columns_not_covered) + 1) - + ind_args_between=np.arange(len(x_ending_all_between_nm_wc)) for column in range(i_s_nc, x_end_biggest_column): ind_args_in_col=ind_args_between[x_starting_all_between_nm_wc==column] @@ -2038,17 +2052,17 @@ def return_boxes_of_images_by_order_of_reading_new( y_lines_by_order.append(y_col_sort[ii]) x_start_by_order.append(x_start_column_sort[ii]) x_end_by_order.append(x_end_column_sort[ii]-1) - + for il in range(len(y_lines_by_order)): y_copy = list(y_lines_by_order) x_start_copy = list(x_start_by_order) x_end_copy = list(x_end_by_order) - + #print(y_copy,'y_copy') y_itself=y_copy.pop(il) x_start_itself=x_start_copy.pop(il) x_end_itself=x_end_copy.pop(il) - + #print(y_copy,'y_copy2') for column in range(x_start_itself, x_end_itself+1): #print(column,'cols') @@ -2065,7 +2079,7 @@ def return_boxes_of_images_by_order_of_reading_new( y_down=np.min(y_in_cols) else: y_down=[int(splitter_y_new[i+1])][0] - #print(y_itself,'y_itself') + #print(y_itself,'y_itself') boxes.append([peaks_neg_tot[column], peaks_neg_tot[column+1], y_itself, @@ -2108,7 +2122,7 @@ def return_boxes_of_images_by_order_of_reading_new( ##x_start_by_order = np.append(x_start_by_order, [0] * len(columns_not_covered)) x_starting = np.append(x_starting, columns_not_covered) x_ending = np.append(x_ending, np.array(columns_not_covered) + 1) - + ind_args=np.array(range(len(y_type_2))) #ind_args=np.array(ind_args) for column in range(len(peaks_neg_tot)-1): @@ -2130,17 +2144,17 @@ def return_boxes_of_images_by_order_of_reading_new( y_lines_by_order.append(y_col_sort[ii]) x_start_by_order.append(x_start_column_sort[ii]) x_end_by_order.append(x_end_column_sort[ii]-1) - + for il in range(len(y_lines_by_order)): y_copy = list(y_lines_by_order) x_start_copy = list(x_start_by_order) x_end_copy = list(x_end_by_order) - + #print(y_copy,'y_copy') y_itself=y_copy.pop(il) x_start_itself=x_start_copy.pop(il) x_end_itself=x_end_copy.pop(il) - + #print(y_copy,'y_copy2') for column in range(x_start_itself, x_end_itself+1): #print(column,'cols') @@ -2157,22 +2171,22 @@ def return_boxes_of_images_by_order_of_reading_new( y_down=np.min(y_in_cols) else: y_down=[int(splitter_y_new[i+1])][0] - #print(y_itself,'y_itself') + #print(y_itself,'y_itself') boxes.append([peaks_neg_tot[column], peaks_neg_tot[column+1], y_itself, y_down]) #else: #boxes.append([ 0, regions_without_separators[:,:].shape[1] ,splitter_y_new[i],splitter_y_new[i+1]]) - - if right2left_readingorder: + + if right2left_readingorder: peaks_neg_tot_tables_new = [] if len(peaks_neg_tot_tables)>=1: for peaks_tab_ind in peaks_neg_tot_tables: peaks_neg_tot_tables_ind = regions_without_separators.shape[1] - np.array(peaks_tab_ind) peaks_neg_tot_tables_ind = list(peaks_neg_tot_tables_ind[::-1]) peaks_neg_tot_tables_new.append(peaks_neg_tot_tables_ind) - + for i in range(len(boxes)): x_start_new = regions_without_separators.shape[1] - boxes[i][1] x_end_new = regions_without_separators.shape[1] - boxes[i][0] diff --git a/src/eynollah/utils/contour.py b/src/eynollah/utils/contour.py index 1adb943..a81ccb4 100644 --- a/src/eynollah/utils/contour.py +++ b/src/eynollah/utils/contour.py @@ -227,9 +227,12 @@ def get_textregion_contours_in_org_image_light_old(cnts, img, slope_first): return cnts_org -def do_back_rotation_and_get_cnt_back(contour_par, index_r_con, img, slope_first): +def do_back_rotation_and_get_cnt_back(contour_par, index_r_con, img, slope_first, confidence_matrix): img_copy = np.zeros(img.shape) img_copy = cv2.fillPoly(img_copy, pts=[contour_par], color=(1, 1, 1)) + + confidence_matrix_mapped_with_contour = confidence_matrix * img_copy[:,:,0] + confidence_contour = np.sum(confidence_matrix_mapped_with_contour) / float(np.sum(img_copy[:,:,0])) img_copy = rotation_image_new(img_copy, -slope_first).astype(np.uint8) imgray = cv2.cvtColor(img_copy, cv2.COLOR_BGR2GRAY) @@ -239,11 +242,13 @@ def do_back_rotation_and_get_cnt_back(contour_par, index_r_con, img, slope_first cont_int[0][:, 0, 0] = cont_int[0][:, 0, 0] + np.abs(img_copy.shape[1] - img.shape[1]) cont_int[0][:, 0, 1] = cont_int[0][:, 0, 1] + np.abs(img_copy.shape[0] - img.shape[0]) # print(np.shape(cont_int[0])) - return cont_int[0], index_r_con + return cont_int[0], index_r_con, confidence_contour -def get_textregion_contours_in_org_image_light(cnts, img, slope_first, map=map): +def get_textregion_contours_in_org_image_light(cnts, img, slope_first, confidence_matrix, map=map): if not len(cnts): - return [] + return [], [] + + confidence_matrix = cv2.resize(confidence_matrix, (int(img.shape[1]/6), int(img.shape[0]/6)), interpolation=cv2.INTER_NEAREST) img = cv2.resize(img, (int(img.shape[1]/6), int(img.shape[0]/6)), interpolation=cv2.INTER_NEAREST) ##cnts = list( (np.array(cnts)/2).astype(np.int16) ) #cnts = cnts/2 @@ -251,10 +256,11 @@ def get_textregion_contours_in_org_image_light(cnts, img, slope_first, map=map): results = map(partial(do_back_rotation_and_get_cnt_back, img=img, slope_first=slope_first, + confidence_matrix=confidence_matrix, ), cnts, range(len(cnts))) - contours, indexes = tuple(zip(*results)) - return [i*6 for i in contours] + contours, indexes, conf_contours = tuple(zip(*results)) + return [i*6 for i in contours], list(conf_contours) def return_contours_of_interested_textline(region_pre_p, pixel): # pixels of images are identified by 5 diff --git a/src/eynollah/utils/rotate.py b/src/eynollah/utils/rotate.py index 603c2d9..189693d 100644 --- a/src/eynollah/utils/rotate.py +++ b/src/eynollah/utils/rotate.py @@ -1,6 +1,4 @@ import math - -import imutils import cv2 def rotatedRectWithMaxArea(w, h, angle): @@ -35,14 +33,14 @@ def rotate_max_area_new(image, rotated, angle): return rotated[y1:y2, x1:x2] def rotation_image_new(img, thetha): - rotated = imutils.rotate(img, thetha) + rotated = rotate_image(img, thetha) return rotate_max_area_new(img, rotated, thetha) def rotate_image(img_patch, slope): (h, w) = img_patch.shape[:2] center = (w // 2, h // 2) M = cv2.getRotationMatrix2D(center, slope, 1.0) - return cv2.warpAffine(img_patch, M, (w, h), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE) + return cv2.warpAffine(img_patch, M, (w, h) ) def rotate_image_different( img, slope): # img = cv2.imread('images/input.jpg') @@ -62,17 +60,17 @@ def rotate_max_area(image, rotated, rotated_textline, rotated_layout, rotated_ta return rotated[y1:y2, x1:x2], rotated_textline[y1:y2, x1:x2], rotated_layout[y1:y2, x1:x2], rotated_table_prediction[y1:y2, x1:x2] def rotation_not_90_func(img, textline, text_regions_p_1, table_prediction, thetha): - rotated = imutils.rotate(img, thetha) - rotated_textline = imutils.rotate(textline, thetha) - rotated_layout = imutils.rotate(text_regions_p_1, thetha) - rotated_table_prediction = imutils.rotate(table_prediction, thetha) + rotated = rotate_image(img, thetha) + rotated_textline = rotate_image(textline, thetha) + rotated_layout = rotate_image(text_regions_p_1, thetha) + rotated_table_prediction = rotate_image(table_prediction, thetha) return rotate_max_area(img, rotated, rotated_textline, rotated_layout, rotated_table_prediction, thetha) def rotation_not_90_func_full_layout(img, textline, text_regions_p_1, text_regions_p_fully, thetha): - rotated = imutils.rotate(img, thetha) - rotated_textline = imutils.rotate(textline, thetha) - rotated_layout = imutils.rotate(text_regions_p_1, thetha) - rotated_layout_full = imutils.rotate(text_regions_p_fully, thetha) + rotated = rotate_image(img, thetha) + rotated_textline = rotate_image(textline, thetha) + rotated_layout = rotate_image(text_regions_p_1, thetha) + rotated_layout_full = rotate_image(text_regions_p_fully, thetha) return rotate_max_area_full_layout(img, rotated, rotated_textline, rotated_layout, rotated_layout_full, thetha) def rotate_max_area_full_layout(image, rotated, rotated_textline, rotated_layout, rotated_layout_full, angle): diff --git a/src/eynollah/writer.py b/src/eynollah/writer.py index 7bcd9af..92e353f 100644 --- a/src/eynollah/writer.py +++ b/src/eynollah/writer.py @@ -139,7 +139,7 @@ class EynollahXmlWriter(): points_co += str(int((contour_textline[0][1] + region_bboxes[0]+page_coord[0])/self.scale_y)) points_co += ' ' coords.set_points(points_co[:-1]) - + def serialize_lines_in_dropcapital(self, text_region, all_found_textline_polygons, region_idx, page_coord, all_box_coord, slopes, counter, ocr_all_textlines_textregion): self.logger.debug('enter serialize_lines_in_region') for j in range(1): @@ -168,7 +168,7 @@ class EynollahXmlWriter(): with open(self.output_filename, 'w') as f: f.write(to_xml(pcgts)) - def build_pagexml_no_full_layout(self, found_polygons_text_region, page_coord, order_of_texts, id_of_texts, all_found_textline_polygons, all_box_coord, found_polygons_text_region_img, found_polygons_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_marginals, cont_page, polygons_lines_to_be_written_in_xml, found_polygons_tables, ocr_all_textlines): + def build_pagexml_no_full_layout(self, found_polygons_text_region, page_coord, order_of_texts, id_of_texts, all_found_textline_polygons, all_box_coord, found_polygons_text_region_img, found_polygons_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_marginals, cont_page, polygons_lines_to_be_written_in_xml, found_polygons_tables, ocr_all_textlines, conf_contours_textregion): self.logger.debug('enter build_pagexml_no_full_layout') # create the file structure @@ -184,8 +184,9 @@ class EynollahXmlWriter(): for mm in range(len(found_polygons_text_region)): textregion = TextRegionType(id=counter.next_region_id, type_='paragraph', - Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_text_region[mm], page_coord)), + Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_text_region[mm], page_coord), conf=conf_contours_textregion[mm]), ) + #textregion.set_conf(conf_contours_textregion[mm]) page.add_TextRegion(textregion) if ocr_all_textlines: ocr_textlines = ocr_all_textlines[mm] @@ -215,9 +216,9 @@ class EynollahXmlWriter(): points_co += ',' points_co += str(int((found_polygons_text_region_img[mm][lmm][1] + page_coord[0])/ self.scale_y )) points_co += ' ' - + img_region.get_Coords().set_points(points_co[:-1]) - + for mm in range(len(polygons_lines_to_be_written_in_xml)): sep_hor = SeparatorRegionType(id=counter.next_region_id, Coords=CoordsType()) page.add_SeparatorRegion(sep_hor) @@ -241,7 +242,7 @@ class EynollahXmlWriter(): return pcgts - def build_pagexml_full_layout(self, found_polygons_text_region, found_polygons_text_region_h, page_coord, order_of_texts, id_of_texts, all_found_textline_polygons, all_found_textline_polygons_h, all_box_coord, all_box_coord_h, found_polygons_text_region_img, found_polygons_tables, found_polygons_drop_capitals, found_polygons_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_h, slopes_marginals, cont_page, polygons_lines_to_be_written_in_xml, ocr_all_textlines): + def build_pagexml_full_layout(self, found_polygons_text_region, found_polygons_text_region_h, page_coord, order_of_texts, id_of_texts, all_found_textline_polygons, all_found_textline_polygons_h, all_box_coord, all_box_coord_h, found_polygons_text_region_img, found_polygons_tables, found_polygons_drop_capitals, found_polygons_marginals, all_found_textline_polygons_marginals, all_box_coord_marginals, slopes, slopes_h, slopes_marginals, cont_page, polygons_lines_to_be_written_in_xml, ocr_all_textlines, conf_contours_textregion, conf_contours_textregion_h): self.logger.debug('enter build_pagexml_full_layout') # create the file structure @@ -256,9 +257,9 @@ class EynollahXmlWriter(): for mm in range(len(found_polygons_text_region)): textregion = TextRegionType(id=counter.next_region_id, type_='paragraph', - Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_text_region[mm], page_coord))) + Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_text_region[mm], page_coord), conf=conf_contours_textregion[mm])) page.add_TextRegion(textregion) - + if ocr_all_textlines: ocr_textlines = ocr_all_textlines[mm] else: @@ -293,10 +294,10 @@ class EynollahXmlWriter(): for mm in range(len(found_polygons_text_region_img)): page.add_ImageRegion(ImageRegionType(id=counter.next_region_id, Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_text_region_img[mm], page_coord)))) - + for mm in range(len(polygons_lines_to_be_written_in_xml)): page.add_SeparatorRegion(ImageRegionType(id=counter.next_region_id, Coords=CoordsType(points=self.calculate_polygon_coords(polygons_lines_to_be_written_in_xml[mm], [0 , 0, 0, 0])))) - + for mm in range(len(found_polygons_tables)): page.add_TableRegion(TableRegionType(id=counter.next_region_id, Coords=CoordsType(points=self.calculate_polygon_coords(found_polygons_tables[mm], page_coord))))