OpenCV OCR: text detection and recognition, tested
1,533 of 1,955
words found by OpenCV alone
Tesseract still reads more words
| Run | Words | Time |
|---|---|---|
| OpenCV alone: EAST + CRNN | 1,533 | 283 s |
| OpenCV boxes + Tesseract | 1,865 | 83 s |
| Plain Tesseract 5.5.4 | 1,878 | 36 s |
EAST and CRNN were made for scene text: a few words in a photo, like a shop sign. On these book pages, plain Tesseract was faster and read more. Times are rough: 2 threads on a shared Mac.
EAST finds the words, CRNN reads them
pip install opencv-python-headless
Then download two model files,
frozen_east_text_detection.pb and
text_recognition_CRNN_CH_2021sep.onnx (links under
Sources). Save this as ocv.py next to them:
import sys
import cv2
cv2.setNumThreads(2)
det = cv2.dnn.TextDetectionModel_EAST("frozen_east_text_detection.pb")
det.setConfidenceThreshold(0.5).setNMSThreshold(0.4)
rec = cv2.dnn.TextRecognitionModel("text_recognition_CRNN_CH_2021sep.onnx")
rec.setDecodeType("CTC-greedy")
rec.setVocabulary(list("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"))
rec.setInputParams(1 / 127.5, (100, 32), (127.5, 127.5, 127.5))
for path in sys.argv[1:]:
img = cv2.imread(path)
h, w = img.shape[:2]
det.setInputParams(1.0, (w // 32 * 32, h // 32 * 32), (123.68, 116.78, 103.94), True)
Sort the boxes into lines
The rest of the page loop:
boxes = sorted((cv2.boundingRect(b) for b in det.detect(img)[0]), key=lambda b: b[1] + b[3] / 2)
lines, top = [], -h
for x, y, bw, bh in boxes:
if y + bh / 2 - top > bh / 2:
lines.append([])
top = y + bh / 2
p = round(bh * 0.15)
lines[-1].append((max(x - p, 0), max(y - p, 0), x + bw + p, y + bh + p))
for line in lines:
print(" ".join(rec.recognize(img[y0:y1, x0:x1]) for x0, y0, x1, y1 in sorted(line)))
Run python ocv.py page.jpg. Padding each box by 15% of
its height found 1,533 words, not 1,392.
Tesseract does best on OpenCV's lines
PP-OCRv3, the OpenCV Zoo detector, boxes whole lines, and Tesseract reads a line well with --psm 7. Swap the detector and the reader, and drop the padding:
import pytesseract
det = cv2.dnn.TextDetectionModel_DB("text_detection_en_ppocrv3_2023may.onnx")
det.setBinaryThreshold(0.3).setPolygonThreshold(0.5).setUnclipRatio(2.0).setMaxCandidates(200)
det.setInputParams(1 / 255, (736, 736), (123.675, 116.28, 103.53), True)
print(" ".join(pytesseract.image_to_string(c, config="--psm 7").strip() for c in crops))
On EAST's word boxes, Tesseract found 1,292 words. Whole lines work better.
The full ocv_tess.py
import sys
import cv2
import pytesseract
cv2.setNumThreads(2)
det = cv2.dnn.TextDetectionModel_DB("text_detection_en_ppocrv3_2023may.onnx")
det.setBinaryThreshold(0.3).setPolygonThreshold(0.5).setUnclipRatio(2.0).setMaxCandidates(200)
det.setInputParams(1 / 255, (736, 736), (123.675, 116.28, 103.53), True)
for path in sys.argv[1:]:
img = cv2.imread(path)
boxes = sorted((cv2.boundingRect(b) for b in det.detect(img)[0]), key=lambda b: b[1] + b[3] / 2)
lines, top = [], -img.shape[0]
for x, y, bw, bh in boxes:
if y + bh / 2 - top > bh / 2:
lines.append([])
top = y + bh / 2
lines[-1].append((max(x, 0), max(y, 0), x + bw, y + bh))
for line in lines:
crops = [img[y0:y1, x0:x1] for x0, y0, x1, y1 in sorted(line)]
print(" ".join(pytesseract.image_to_string(c, config="--psm 7").strip() for c in crops))
EAST at 320x320 misses most words
The OpenCV tutorial gives EAST a 320x320 input. On a book page that shrinks the text to a few pixels: with Tesseract on each box, 105 of 1,955 words.
Give EAST the page size, cut down to a multiple of 32, as ocv.py does:
(w // 32 * 32, h // 32 * 32)
The zoo pair reads 3 words
The OpenCV Zoo pairs PP-OCRv3 with CRNN. CRNN squeezes each line box into 100 by 32 pixels, and its alphabet has no space.
The running head, The Boy Scouts of the Air, came out as Nerhig-honiogie.ds. CRNN_CH will not take a wider input, as (200, 32):
layers/reshape2_layer.cpp:179: error: (-215:Assertion failed) outTotal == inpTotal in function 'getOutShape'
The 36-character model loses capitals
| Model | Exact | Case and punctuation ignored |
|---|---|---|
| CRNN_CH, 94 characters | 1,533 | 1,679 |
| CRNN_EN, 36 characters | 1,240 | 1,544 |
My score counts a word only with the same case and punctuation. CRNN_EN outputs only digits and lowercase letters, so 599 of the words are out of its reach.
Errors, and mistakes with no error
| Error | Fix |
|---|---|
| Inconsistent shape for ConcatLayer | Make the input size a multiple of 32. |
| Input size not specified | Call setInputParams on the detector. |
| vocabulary is not specified | Call setVocabulary with the model's characters. |
| Failed to parse ONNX model | Check for a Git LFS pointer file. Get the real file. |
No error, wrong result:
- No setInputParams on the recognizer: Keep came out as l0n.
- The 36-character list on the 94-character model: Keep became keep.
- Scale 1.0 on PP-OCRv3: 2 boxes on page one, not 29.
The full error lines
Each OpenCV line starts with OpenCV(5.0.0) and a long build path.
EAST at (1920, 3079):
layers/concat_layer.cpp:110: error: (-201:Incorrect size of input array) Inconsistent shape for ConcatLayer in function 'getMemoryShapes'
PP-OCRv3 at (1920, 3079):
layers/concat2_layer.cpp:127: error: (-215:Assertion failed) inpShape0[j] == inpShape_i[j] in function 'getOutShape'
No setInputParams:
model.cpp:115: error: (-201:Incorrect size of input array) Input size not specified in function 'processFrame'
No setVocabulary:
model.cpp:720: error: (-5:Bad argument) TextRecognitionModel: vocabulary is not specified in function 'decode'
No setDecodeType:
model.cpp:718: error: (-5:Bad argument) TextRecognitionModel: decodeType is not specified in function 'decode'
A Git LFS pointer file:
onnx/onnx_importer2.cpp:346: error: (-210:Unsupported format or combination of formats) Failed to parse ONNX model: lfs/text_detection_en_ppocrv3_2023may.onnx in function 'parseFile'
SystemError: <class 'cv2.dnn.TextDetectionModel_DB'> returned a result with an exception set
How I tested it
- Pages: the 10 scans of OCR in n8n, proofread Wikisource book pages, 1,955 true words, as 1920 px color JPEGs.
- OpenCV 5.0.0 (opencv-python-headless 5.0.0.93), pytesseract 0.3.13 and Tesseract 5.5.4, on an Apple Silicon Mac, on the CPU.
- Settings from the OpenCV tutorial. EAST: confidence 0.5, NMS 0.4. PP-OCRv3: 736x736, as in the OpenCV Zoo. CRNN: 100x32.
- One run at a time, lowest priority, 2 threads. Other jobs shared the Mac, so times are rough.
- Score: true words found in order, with case and punctuation.
What I did not test
- The tutorial's DB_IC15 and DB_TD500 models: their source, MhLiao/DB, has no license, so I did not run them.
- Photos, signs and screens: I tested scanned book pages only.
- A GPU, the C++ and JavaScript APIs, and the int8 models.
Sources
Scanned pages to text, no models
Run it yourself. Our Apify Actor reads PDF, PNG, JPEG, WEBP and TIFF for $0.006 a page, with no model files. Its code found 1,879 of these 1,955 words.
Or send us the scan. Tack, our AI, does the job for $29, up to 50 pages. First page free.
Questions people ask
Can OpenCV do OCR without Tesseract?
Yes. EAST finds the words and CRNN reads them, both in cv2.dnn. On 10 scanned book pages they found 1,533 of 1,955 words. Plain Tesseract found 1,878.
Why does EAST text detection miss most of my page?
The tutorial's 320x320 input shrinks a full page until the text is a few pixels tall. With Tesseract on each box that found 105 of 1,955 words. Give EAST the page size cut down to a multiple of 32: (w // 32 * 32, h // 32 * 32).
Why does TextRecognitionModel return lowercase text?
The model or the vocabulary has 36 characters: digits and a to z. For capitals and punctuation, use text_recognition_CRNN_CH_2021sep.onnx with its 94-character list.