Training continues endlessly

[SigawathRungrueng]

Training continues endlessly with only 625 images.
During training, it uses 100% GPU utilization but doesn't use any VRAM.

Here is my config,

Character Sets

number: '0123456789'
symbol: "!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ €"
lang_char: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZกขคฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรลวศษสหฬอฮฤเแโใไะาำํฺฯๆ0123456789๑๒๓๔๕๖๗๘๙'

Experiment Setup

experiment_name: 'filtered_short_Thai'
train_data: 'all_data3'
valid_data: 'all_data3/TH_val1'
manualSeed: 1111
saved_model: 'saved_models/thai.pth'

Training Loop Settings

workers: 12
batch_size: 32
num_iter: 1
valInterval: 1
FT: True

OptimizerI ne

optim: 'Adam'
lr: 0.0001
beta1: 0.9
rho: 0.95
eps: 0.0000001
grad_clip: 5

Data Settings

select_data: 'en_train'
batch_ratio: '1'
total_data_usage_ratio: 1.0
batch_max_length: 25
imgH: 48
imgW: 256
rgb: False
contrast_adjust: 0.0
sensitive: True
PAD: True
data_filtering_off: False

Model Architecture

Transformation: 'None'
SequenceModeling: 'None'
Prediction: 'CTC'

Model Hyperparameters

num_fiducial: 20
input_channel: 1
output_channel: 512
hidden_size: 512

Decoding Strategy

decode: 'greedy'
new_prediction: True
freeze_FeatureFxtraction: False
freeze_SequenceModeling: False

Could you please help fix this?

[populated]

"Please do not create an issue about data generation and model training in this repository. If you have any question regarding data generation and model training, please ask in the respective repositories."

1. https://github.com/Belval/TextRecognitionDataGenerator
2. https://github.com/clovaai/deep-text-recognition-benchmark