This tool performs document layout analysis (segmentation) from image data and returns the results as [PAGE-XML](https://github.com/PRImA-Research-Lab/PAGE-XML).
:warning: Development is currently focused on achieving the best possible quality of results for a wide variety of historical documents and therefore processing can be very slow. We aim to improve this, but contributions are welcome.
It can currently detect the following layout classes/elements:
In addition, the tool can be used to detect the _[ReadingOrder](https://ocr-d.de/en/gt-guidelines/trans/lyLeserichtung.html)_ of regions. The final goal is to feed the output to an OCR model.
Python `3.8-3.11` with Tensorflow `2.12-2.15` on Linux are currently supported.
For (limited) GPU support the CUDA toolkit needs to be installed.
The tool uses a combination of various models and heuristics (see flowchart below for the different stages and how they interact):
or clone the repository, enter it and install (editable) with
## Border detection
For the purpose of text recognition (OCR) and in order to avoid noise being introduced from texts outside the printspace, one first needs to detect the border of the printed frame. This is done by a binary pixel-wise-segmentation model trained on a dataset of 2,000 documents where about 1,200 of them come from the [dhSegment](https://github.com/dhlab-epfl/dhSegment/) project (you can download the dataset from [here](https://github.com/dhlab-epfl/dhSegment/releases/download/v0.2/pages.zip)) and the remainder having been annotated in SBB. For border detection, the model needs to be fed with the whole image at once rather than separated in patches.
```
## Layout detection
git clone git@github.com:qurator-spk/eynollah.git
As a next step, text regions need to be identified by means of layout detection. Again a pixel-wise segmentation model was trained on 131 labeled images from the SBB digital collections, including some data augmentation. Since the target of this tool are historical documents, we consider as main region types text regions, separators, images, tables and background - each with their own subclasses, e.g. in the case of text regions, subclasses like header/heading, drop capital, main body text etc. While it would be desirable to detect and classify each of these classes in a granular way, there are also limitations due to having a suitably large and balanced training set. Accordingly, the current version of this tool is focussed on the main region types background, text region, image and separator.
cd eynollah; pip install -e .
```
## Textline detection
In a subsequent step, binary pixel-wise segmentation is used again to classify pixels in a document that constitute textlines. For textline segmentation, a model was initially trained on documents with only one column/block of text and some augmentation with regard to scaling. By fine-tuning the parameters also for multi-column documents, additional training data was produced that resulted in a much more robust textline detection model.
## Image enhancement
This is an image to image model which input was low quality of an image and label was actually the original image. For this one we did not have any GT, so we decreased the quality of documents in SBB and then feed them into model.
## Scale classification
This is simply an image classifier which classifies images based on their scales or better to say based on their number of columns.
## Heuristic methods
Some heuristic methods are also employed to further improve the model predictions:
* After border detection, the largest contour is determined by a bounding box, and the image cropped to these coordinates.
* For text region detection, the image is scaled up to make it easier for the model to detect background space between text regions.
* A minimum area is defined for text regions in relation to the overall image dimensions, so that very small regions that are noise can be filtered out.
* Deskewing is applied on the text region level (due to regions having different degrees of skew) in order to improve the textline segmentation result.
* After deskewing, a calculation of the pixel distribution on the X-axis allows the separation of textlines (foreground) and background pixels.
* Finally, using the derived coordinates, bounding boxes are determined for each textline.
## Installation
`pip install .` or
Alternatively, you can run `make install` or `make install-dev` for editable installation.
`pip install . -e` for editable installation
## Models
Alternatively, you can also use `make` with these targets:
Pre-trained models can be downloaded from [qurator-data.de](https://qurator-data.de/eynollah/) or [huggingface](https://huggingface.co/SBB?search_models=eynollah).
## Train
`make install` or
🚧 **Work in progress**
In case you want to train your own model, have a look at [`sbb_pixelwise_segmentation`](https://github.com/qurator-spk/sbb_pixelwise_segmentation).
`make install-dev` for editable installation
### Models
In order to run this tool you also need trained models. You can download our pretrained models from [qurator-data.de](https://qurator-data.de/eynollah/).
Alternatively, running `make models` will download and extract models to `$(PWD)/models_eynollah`.
## Usage
## Usage
The command-line interface can be called like this:
The basic command-line interface can be called like this:
| `-fl` | full layout analysis including all steps and segmentation classes |
| `-light` | lighter and faster but simpler method for main region detection and deskewing |
| `-tab` | apply table detection |
| `-ae` | apply enhancement (the resulting image is saved to the output directory) |
| `-as` | apply scaling |
| `-cl` | apply contour detection for curved text lines instead of bounding boxes |
| `-ib` | apply binarization (the resulting image is saved to the output directory) |
| `-ep` | enable plotting (MUST always be used with `-sl`, `-sd`, `-sa`, `-si` or `-ae`) |
| `-eoi` | extract only images to output directory (other processing will not be done) |
| `-ho` | ignore headers for reading order dectection |
| `-si <directory>` | save image regions detected to this directory |
| `-sd <directory>` | save deskewed image to this directory |
| `-sl <directory>` | save layout prediction as plot to this directory |
| `-sp <directory>` | save cropped page image to this directory |
| `-sa <directory>` | save all (plot, enhanced/binary image, layout) to this directory |
If no option is set, the tool performs layout detection of main regions (background, text, images, separators and marginals).
Here are the difference in elements detected depending on the `--full-layout`/`--no-full-layout` command line flags:
The best output quality is produced when RGB images are used as input rather than greyscale or binarized images.
#### Use as OCR-D processor
| | `--full-layout` | `--no-full-layout` |
🚧 **Work in progress**
| --- | --- | --- |
| reading order | x | x |
| header regions | x | - |
| text regions | x | x |
| text regions / text line | x | x |
| drop-capitals | x | - |
| marginals | x | x |
| marginals / text line | x | x |
| image region | x | x |
Eynollah ships with a CLI interface to be used as [OCR-D](https://ocr-d.de) processor.
### How to use
In this case, the source image file group with (preferably) RGB images should be used as input like this:
First, this model makes use of up to 9 trained models which are responsible for different operations like size detection, column classification, image enhancement, page extraction, main layout detection, full layout detection and textline detection.That does not mean that all 9 models are always required for every document. Based on the document characteristics and parameters specified, different scenarios can be applied.
```
* If none of the parameters is set to `true`, the tool will perform a layout detection of main regions (background, text, images, separators and marginals). An advantage of this tool is that it tries to extract main text regions separately as much as possible.
Any image referenced by `@imageFilename` in PAGE-XML is passed on directly to Eynollah as a processor, so that e.g.
* If you set `-ae` (**a**llow image **e**nhancement) parameter to `true`, the tool will first check the ppi (pixel-per-inch) of the image and when it is less than 300, the tool will resize it and only then image enhancement will occur. Image enhancement can also take place without this option, but by setting this option to `true`, the layout xml data (e.g. coordinates) will be based on the resized and enhanced image instead of the original image.
```
* For some documents, while the quality is good, their scale is very large, and the performance of tool decreases. In such cases you can set `-as` (**a**llow **s**caling) to `true`. With this option enabled, the tool will try to rescale the image and only then the layout detection process will begin.
uses the original (RGB) image despite any binarization that may have occured in previous OCR-D processing steps
* If you care about drop capitals (initials) and headings, you can set `-fl` (**f**ull **l**ayout) to `true`. With this setting, the tool can currently distinguish 7 document layout classes/elements.
#### Additional documentation
* In cases where the document includes curved headers or curved lines, rectangular bounding boxes for textlines will not be a great option. In such cases it is strongly recommended setting the flag `-cl` (**c**urved **l**ines) to `true` to find contours of curved lines instead of rectangular bounding boxes. Be advised that enabling this option increases the processing time of the tool.
Please check the [wiki](https://github.com/qurator-spk/eynollah/wiki).
* To crop and save image regions inside the document, set the parameter `-si` (**s**ave **i**mages) to true and provide a directory path to store the extracted images.
## How to cite
If you find this tool useful in your work, please consider citing our paper:
* This tool is actively being developed. If problems occur, or the performance does not meet your expectations, we welcome your feedback via [issues](https://github.com/qurator-spk/eynollah/issues).
```bibtex
@inproceedings{hip23rezanezhad,
title = {Document Layout Analysis with Deep Learning and Heuristics},
author = {Rezanezhad, Vahid and Baierer, Konstantin and Gerber, Mike and Labusch, Kai and Neudecker, Clemens},
booktitle = {Proceedings of the 7th International Workshop on Historical Document Imaging and Processing {HIP} 2023,
San José, CA, USA, August 25-26, 2023},
publisher = {Association for Computing Machinery},
help="If a directory is given, only images in documents will be cropped and saved there and the other processing will not be done",
)
@click.option(
@click.option(
"--allow-enhancement/--no-allow-enhancement",
"--allow-enhancement/--no-allow-enhancement",
"-ae/-noae",
"-ae/-noae",
@ -83,13 +65,7 @@ from eynollah.eynollah import Eynollah
"--curved-line/--no-curvedline",
"--curved-line/--no-curvedline",
"-cl/-nocl",
"-cl/-nocl",
is_flag=True,
is_flag=True,
help="if this parameter set to true, this tool will try to return contoure of textlines instead of rectangle bounding box of textline. This should be taken into account that with this option the tool need more time to do process.",
help="if this parameter set to true, this tool will try to return contoure of textlines instead of rectabgle bounding box of textline. This should be taken into account that with this option the tool need more time to do process.",
)
@click.option(
"--textline_light/--no-textline_light",
"-tll/-notll",
is_flag=True,
help="if this parameter set to true, this tool will try to return contoure of textlines instead of rectangle bounding box of textline with a faster method.",
)
)
@click.option(
@click.option(
"--full-layout/--no-full-layout",
"--full-layout/--no-full-layout",
@ -97,18 +73,6 @@ from eynollah.eynollah import Eynollah
is_flag=True,
is_flag=True,
help="if this parameter set to true, this tool will try to return all elements of layout.",
help="if this parameter set to true, this tool will try to return all elements of layout.",
)
)
@click.option(
"--tables/--no-tables",
"-tab/-notab",
is_flag=True,
help="if this parameter set to true, this tool will try to detect tables.",
)
@click.option(
"--right2left/--left2right",
"-r2l/-l2r",
is_flag=True,
help="if this parameter set to true, this tool will extract right-to-left reading order.",
)
@click.option(
@click.option(
"--input_binary/--input-RGB",
"--input_binary/--input-RGB",
"-ib/-irgb",
"-ib/-irgb",
@ -122,23 +86,11 @@ from eynollah.eynollah import Eynollah
help="if this parameter set to true, this tool would check the scale and if needed it will scale it to perform better layout detection",
help="if this parameter set to true, this tool would check the scale and if needed it will scale it to perform better layout detection",
)
)
@click.option(
@click.option(
"--headers_off/--headers-on",
"--headers-off/--headers-on",
"-ho/-noho",
"-ho/-noho",
is_flag=True,
is_flag=True,
help="if this parameter set to true, this tool would ignore headers role in reading order",
help="if this parameter set to true, this tool would ignore headers role in reading order",
)
)
@click.option(
"--light_version/--original",
"-light/-org",
is_flag=True,
help="if this parameter set to true, this tool would use lighter version",
print('Error: You used -eoi which can not be enabled alongside light_version -light or allow_scaling -as or allow_enhancement -ae or curved_line -cl or textline_light -tll or full_layout -fl or tables -tab or right2left -r2l or headers_off -ho')
print("Error: You used -ep to enable plotting but set none of -sl, -sd, -sa or -si")
# areas_main=np.array([cv2.contourArea(all_found_textline_polygons[int(region_final)][0][j] ) for j in range(len(all_found_textline_polygons[int(region_final)]))])
# areas_main=np.array([cv2.contourArea(all_found_texline_polygons[int(region_final)][0][j] ) for j in range(len(all_found_texline_polygons[int(region_final)]))])
# areas_main=np.array([cv2.contourArea(all_found_textline_polygons[int(region_final)][0][j] ) for j in range(len(all_found_textline_polygons[int(region_final)]))])
# areas_main=np.array([cv2.contourArea(all_found_texline_polygons[int(region_final)][0][j] ) for j in range(len(all_found_texline_polygons[int(region_final)]))])