A very quick and dirty text reading difficulty analysis library

This library provides a few different algorithms for determining the difficulty to read of a certain text. It is not accurate enough to be used in research, it's a good estimation tool, nothing more. For basically every index/level that exists you want to aim for 7-10 in any non-academic work intended for adults, though this is not a hard rule. It is also single-threaded, meaning it's quite slow on large text. This will be addressed at some point when I get time.

The effectiveness of this library is massively limited by it's accuracy in:

• counting sentences; This also uses a heuristic since text often breaks the standard grammatical formatting. This also makes all of the measurements worse at text like poetry, since sentences often don't end with punctuation.
• counting syllables; A heuristic is used to calculate this since there's no known perfect way of doing it

The error percentages go up the more the text is not just in simple plaintext format with standard sentence formatting (i.e. ending sentences with proper punctuation). Likewise there are major limitations in what types of text are possible to analyze:

• Unicode characters are completely ignored
• Non-english text with english characters will be incorrectly parsed
• Numbers are ignored
• Non-sentence text data will cause errors (i.e. math formulas in markdown text)

You can add the package to your project using:

go get https://github.com/Descent098/readn

This method is the most variant of the three (effected by error rates), but typically the most widely used in literature. It tends to UNDERESTIMATE (lower level, higher ease), level tends to be 0.5-2 years less than it should be depending on formatting, and ease tends to be 1-2 bands higher than it should be (i.e. you may get 40 putting it at a college level, but it likely should be College graduate or proffessional). The method returns 2 values the FleschKincaidResult.Ease and the FleschKincaidResult.Level. The Level is essentially the numebr of years of education to understand, the ease is the opposite, it's a score where the higher the ease, the easier the text is to read. Ease is approximately:

package main

import "https://github.com/Descent098/readn"

func main(){
    text := `some text here`

    res:= readn.FleschKincaid(text)

    fmt.Printf("Your Kincaid ease is ~%.2f your education index it ~%.2f years education", res.Ease, res.Level)
}

This is the algorithm is a good alternative to Flesh-Kincaid for business use cases. The index is basically index-1 years of education to read. So

ARI is the most accurate in this package because it does not rely on sylables, opting for character count, which is trivial to do. This means the error % is just dependent on sentence count accuracy. It is also designed specifically for technical documents, and works best in this context, though it also works for more general use cases.

package main

import "https://github.com/Descent098/readn"

func main(){
    text := `some text here`

    index := readn.AutomatedReadabilityIndex(text)

    fmt.Printf("Your ARI is ~%.2f years education", index)
}

$4.71(\frac{\textbf{characters}}{\textbf{words}})+0.5(\frac{\textbf{words}}{sentences})-21.43$

SMOG tends to be best suited for the medical field, and is the recommended choice for most paper publishers. It is designed primarily for Medical writing, but it does work outside this context. However it's important to note it does not work on text with less than 30 sentences. The number given is essentially the number of years of education you should have to be able to read the text.

package main

import "https://github.com/Descent098/readn"

func main(){
    text := `some text here`

    val, err := readn.SimpleMeasureOfGobbledygook(text)

    if err !=nil{
        fmt.Fatal("Text was too short for SMOG analysis")
    }

    fmt.Printf("Your SMOG score is ~%.2f years education", index)
}

$grade=1.0430 \sqrt{\textbf{number of polysyllabic words} \times \frac{30}{\textbf{number of sentences}}} +3.1291$

A polysyllabic word is defined as any word with 3 or more syllables

$ease=206.835-1.015(\frac{\textbf{total words}}{\textbf{total sentences}})-84.6(\frac{\textbf{total syllables}}{\textbf{total words}})$

$level=0.39(\frac{\textbf{total words}}{\textbf{total sentences}})+11.8(\frac{\textbf{total syllables}}{\textbf{total words}})-15.59$

• https://en.wikipedia.org/wiki/SMOG
• https://en.wikipedia.org/wiki/Automated_readability_index
• https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests

Testing data is taken from:

https://www.gutenberg.org/ and requires the inclusion of the following notice in relation to each file in /testdata:

This eBook is for the use of anyone anywhere in the United States and most
other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this eBook or online
at www.gutenberg.org. If you
are not located in the United States, you will have to check the laws
of the country where you are located before using this eBook.

Additionally reference values to test against were adapted from the following sources:

• corgis
  • corgis classics
• https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests