Automating YouTube Uploads With OCR Part 5: Refinements and Improving Accuracy

Having limited output possibilities helps immensely

We’ve been using pytesseract to help us OCR screen in Deep Rock Galactic to get metadata for YouTube uploads.

Last time we explored a number of approaches to get the output on the right track. We settled on using a second image from the end screen which had clearer text to augment the processing.

Colour Inversion

Let’s see if we can improve that further with box refinements and what the tesseract wiki suggests.

Yes:

             file                          names             mission_type                       biome      hazard        mission_name                      minerals
0  tests/drg1.png   [graham, MaoTheCat, BertieB]               1 EGG HUNT                  MAGMA CORE  HAZARD 3 -          OPEN TRICK       .ENDH PEARL UMANITE\n98
1  tests/drg2.png                  [&l, [T, @&3]      > miNiNG ExPeDITIBN  RADIOACTIVE EXCLUSION ZONE  HAZARD 3 -     PURIFIED LEGACY         ‘ MAGNITE UMANITE\n17
2  tests/drg3.png       [BertieB, L), MaoTheCat]        MINING EXPEDITION              GLACIAL STRATA  HAZARD 3 -     UNHEALTHY WRECK          ‘ MAGNITE CROPPA\n41
3  tests/drg4.png               [T, 3 Oz!, o\no]         ALVAGE OPERATION                  MAGMA CORE    HAZARD 4        RAPID POCKET      BISMOR ENOR PEARL\n22 24
4  tests/drg5.png                [o383, (o383, ]       ~ POINT EXTRACTION                    SALTPITS    HAZARD 4          ANGRY LUCK         BISMOR UMANITE\n94 19
5  tests/drg6.png  [BertieB, Costello, Noobface]        SALVAGE OPERATION               DENSE BIOZONE    HAZARD 4      RANGER'S PRIZE             ‘ CROPPA JADIZ\n8
6  tests/drg7.png            [®29, @&28, T VL R]                | EGGHUNT                 FUNGUS BOGS    HAZARD 4     CECOND COMEBACK             ‘BISHUH JADIZ\na8
7  tests/drg8.png         [IR A )], Costello, T]  y\n\n MINING EXPEDITION              GLACIAL STRATA    HAZARD 4       COLDSSAL DOOM  ‘ UMANITE ENOR PEARL\n169 48
8  tests/drg9.png             [. ®29, (o], I ‘4]           EGG HUNT __ .l  RADIOACTIVE EXCLUSION ZONE    HAZARD 4  ILLUMINATED POCKET     .ENDH PEARL I MAGNITE\n29

Inverting the image to be black-on-white helps hugely. In fact, given many of the fields have very restricted possibilities, we probably have enough to work with, once we take care of variable number of names.

Handling Different Numbers of Players / Names

In DRG there are 1-4 players. My games are usually 3 or 4 players, sometimes 2, very very rarely solo. As the players names appear in different positions depending on the number of players we need to either

i) use fixed boxes for each number and see which one has sensible output

ii) use OpenCV to detect text to OCR

The first way is manageable in a relatively straightforward manner. Since there is a small number of regular players including myself, we can check for the presence of any of those in the output and keep it if it seems sensible.

Doing that gets us to:

There’s a bit of overdetection, particularly in the last row, which actually only had two players. We can clean things up by:

i) if a name is BertieB with anything else, it’s BertieB as my name doesn’t change (Note this may not be true for everyone- some folks like to change their username)

ii) non-alphanumeric names can be pruned

iii) names of 1-3 chars are likely noise detected as text*

* The last one could probably be dealt with by appropriate thresholding, but that’s a topic for another time.

Doing that, we get:

Which is a huge improvement. We could hard-lock the output to a subset of names (which 99% of my games are with), but that would be a headache to remember to check in the case of playing a game on a public server or people who want to join in my stream. This is “good enough” for the time being!

Levenshtein Distance

Using the Levenshtein distance – the number of edits needed to transform a string into another – we can compare the OCR’d text to the five mission types, and pick whichever is closest. We can do the same thing with the biomes, minerals, and mission names. It should work excellently for the first three as there are few choices; however it should still work acceptably well for the mission names, even though there are over a hundred first at last names.

Our code is simple:

def hard_cast_text(detected_text, choices):                                                                       
      """Hard cast detected_text to one of list of choices"""                                                       
      from Levenshtein import distance                                                                              
      distances = {}                                                                                                
                                                                                                                    
      for choice in choices:                                                                                        
          distances[choice] = distance(choice,lower(),                                                              
                                       detected_text.lower())                                                       
                                                                                                                    
      return min(distances, key=distances.get)

This could probably be made a one-liner if I thought long and hard enough about it. But we’re here to automate, not golf python.

The minerals needed a little extra to handle enor pearl being two words and certain detections being closer in Levenshtein distance to, say, jadiz. Another scoring system that weights the beginning of strings more heavily may have helped there, but keeping it to Levenshtein means I can strip out the external library and implement my own if I so wish.

Our output for these nine tests looks good:

             file                                  names       mission_type                       biome    hazard        mission_name               minerals
0  tests/drg1.png           [graham, MaoTheCat, BertieB]           Egg Hunt                  Magma Core  Hazard 3          Open Trick  [Umanite, Enor Pearl]
1  tests/drg2.png  [BertieB, graham, MaoTheCat, ksyme99]  Mining Expedition  Radioactive Exclusion Zone  Hazard 3     Purified Legacy     [Magnite, Umanite]
2  tests/drg3.png           [BertieB, graham, MaoTheCat]  Mining Expedition              Glacial Strata  Hazard 3     Unhealthy Wreck      [Croppa, Magnite]
3  tests/drg4.png                    [BertieB, Costello]  Salvage Operation                  Magma Core  Hazard 4        Rapid Pocket   [Bismor, Enor Pearl]
4  tests/drg5.png  [BertieB, graham, Noobface, Costello]   Point Extraction                   Salt Pits  Hazard 4          Angry Luck      [Bismor, Umanite]
5  tests/drg6.png          [BertieB, Costello, Noobface]  Salvage Operation               Dense Biozone  Hazard 4      Ranger's Prize        [Jadiz, Croppa]
6  tests/drg7.png           [BertieB, Costello, bTRRABN]           Egg Hunt                 Fungus Bogs  Hazard 4     Second Comeback        [Bismor, Jadiz]
7  tests/drg8.png            [BertieB, Costello, graham]  Mining Expedition              Glacial Strata  Hazard 4       Colossal Doom  [Umanite, Enor Pearl]
8  tests/drg9.png                    [BertieB, Costello]           Egg Hunt  Radioactive Exclusion Zone  Hazard 4  Illuminated Pocket  [Magnite, Enor Pearl]

Next step? Further automation, of course!

Tell us what's on your mind