Japanese Search problem in 5.2.3

[peaceiris]

Description

We cannot search in Japanese with 5.2.3

Expected behavior

We can search in Japanese.

Actual behavior

We cannot search in Japanese.

Steps to reproduce the bug

• With 5.2.2, it works well
• With 5.2.3, it has the problem

git clone https://github.com/SI-Aizu/documentation.git
cd ./documentation

# 5.2.3
git checkout e55dc69
docker-compose up
# Go to http://localhost:8888
# Search "メンバー", it fails.

# 5.2.2
git checkout 6e359cf
docker-compose up
# Go to http://localhost:8888
# Search "メンバー", it will be found.

I am using my Docker image which is managed at peaceiris/mkdocs-material-boilerplate.

Package versions

• Python: 3.8
• MkDocs: 1.1.2
• Material: 5.2.3

Project configuration

https://github.com/SI-Aizu/documentation/blob/6e359cf037eb939bde922b677257aecbb18390ef/mkdocs.yml

# Project information
site_name: 'SI-Aizu Documentation'
site_description: 'SI-Aizu Documentation'
site_author: 'SI-Aizu'
site_url: 'https://si-aizu.github.io/documentation'

# Copyright
copyright: '© 2019 SI-Aizu'

# Repository
repo_name: 'GitHub'
repo_url: 'https://github.com/SI-Aizu/documentation'
edit_uri: 'https://github.com/SI-Aizu/documentation/edit/master/docs'

# Configuration
docs_dir: 'docs'
theme:
  name: 'material'
  language: 'ja'
  shortcuts:
    help: 191    # ?
    next: 78     # n
    previous: 80 # p
    search: 83   # s
  palette:
    primary: indigo
    accent: orange
  # features:
  #   - tabs
  #font:
    #text: 'Ubuntu'
    #code: 'Ubuntu Mono'
  #icon:
    #logo: 'images/logo.svg'
  #favicon: 'assets/images/favicon.ico'

# Extensions
markdown_extensions:
  - admonition
  - codehilite
  - footnotes
  - mdx_math:
      enable_dollar_delimiter: true
  - toc:
      permalink: true
      separator: '_'

plugins:
  - search:
      separator: '[\s\-\.]+'
      lang:
        - ja

# Customization
extra_javascript:
  - 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.6/MathJax.js?config=TeX-MML-AM_CHTML'
extra_css:
  - 'assets/css/custom.css'
extra:
  # manifest: 'manifest.json'
  social:
    - icon: 'fontawesome/brands/github-alt'
      link: 'https://github.com/SI-Aizu/documentation'

System information

• OS: macOS, Linux (Ubuntu 18.04)
• Browser: Google Chrome

[squidfunk]

Possibly related to #1745 (comment)

[squidfunk]

This is trickier than expected. I'll try to find some more time to debug this on the weekend.

In the meantime, you can use 5.2.2.

[squidfunk]

I took this opportunity to revisit the multilingual search functionality. I've found that we didn't quite correctly initialize the pipeline, which is now fixed in e6e7536. This leads to much better type-ahead behavior in many languages. I tested all of them, which took me quite a significant amount of time.

In general, we have two cases:

1. Languages for which lunr-languages provides support, like de, es, etc. These languages are configured to use the language-specific pipeline. The stemmer is removed, as it cripples type-ahead experience.

2. Languages for which we don't have dedicated support. This is the majority. Most of the languages work really well with the default configuration when removing the stemmer, for example pl. Some languages, like kr don't, but removing the entire pipeline seems to work well for them.

Which leads me to the issue you reported. The search terms you provided now works again:

However, there may be some combinations of Japanese characters that don't work, as was shown from my testing. This, however, appears to be an issue with lunr-languages and not Material for MkDocs. If you wish to follow up on this problem, I suggest you open an issue on the lunr-languages repository. Other than that, this issue is considered fixed.

I'll issue a release shortly.

[squidfunk]

Released as part of 5.3.2.

[peaceiris]

I tested the release 5.3.2. The problem was fixed. Thank you very much.

However, there may be some combinations of Japanese characters that don't work, as was shown from my testing. This, however, appears to be an issue with lunr-languages and not Material for MkDocs. If you wish to follow up on this problem, I suggest you open an issue on the lunr-languages repository. Other than that, this issue is considered fixed.

Yes. I'll do it.