[Feature]: Issue 45 | Content Classification (Contextual Tagging)

[TylerB24890]

What?

Closes #45

This PR introduces Content Classification (Contextual Tagging) to the AI Experiments plugin. It allows for one-click AI-powered suggestions for post tags and categories based on a comprehensive analysis of the post's content (as well as title and excerpt). It helps authors apply a relevant and consistent taxonomy, improving content organization, discoverability, and on-site navigation.

Why?

Properly categorizing and tagging content is fundamental to an effective content strategy. It helps readers find related articles and signals relevance to search engines. However, this process is often inconsistent and subjective. This feature removes the guesswork by providing intelligent, context-aware suggestions, leading to a better-organized and more interconnected website.

How?

• Registered a new ContentClassification Experiment.
• Registered new settings to enable Content Classification + determine the strategy (Suggest New, Use Existing) and max suggestions.
• Hook into the block editor editor.PostTaxonomyType and inject the Experiment UI (buttons, suggestions)
  • Opt'd for UI built-in to the existing term selectors rather than providing a modal experience as mocked-up in the issue description. See screenshots below for further details.
• Apply a custom system prompt to ensure the experiment settings are respected (new terms, existing terms, max terms, etc...)
• Various filters are available to customize the output;
  • wpai_content_classification_content - Filter content before AI processing
  • wpai_content_classification_suggestions - Filter suggestions after AI processing
  • wpai_content_classification_strategy - Filter the strategy setting
  • wpai_content_classification_max_suggestions - Filter the max suggestions setting

Use of AI Tools

• AI Assistance: Yes
• Tool(s): Claude Code (Opus 4.6)
• Used for:
  • Documentation review and workflow summary
  • Experiment scaffolding
  • Unit test boilerplates
  • Debugging support & code suggestions
  • Inline documentation and experiment documentation generation
  • Assistance with the system prompt generation

Testing Instructions

1. Navigate to the Experiments page and enable the Content Classification experiment. Choose the strategy and maximum suggestions.
2. Navigate to the block editor the core post post type and confirm the Suggest Categories and Suggest Tags buttons are available in the term selector panels.
3. Click one of the buttons and confirm terms are returned from the AI according to your selected preferences. If "Existing terms only" is selected, your response should only consist of existing terms with no new terms suggested.
   • If no terms are available in the taxonomy an error message will appear at the top of the block editor.
4. Click on the returned terms and confirm they are assigned (or created + assigned) as expected.
5. Navigate back to the Experiments settings page and change the settings for the Content Classification experiment.
   • If you previously suggested existing terms, change it to the "New Terms" option.
   • Increase/decrease the maximum suggestions.
6. Navigate back to the block editor and repeat steps 3 & 4, confirming the results have changed according to your selected preferences.

Screenshots or screencast

Settings UI:

Suggestion UI (block editor):

New Term Suggestions:

Existing Term Suggestions:

Additional Notes

• Various features from the issue discussion were put on hold for the initial PR. These features can be added in a new PR or after the initial PR review.
  • Empty taxonomy callout - Proactive "Need some tags? Get AI suggestions" prompt when post has 150+ words but no tags or categories.
  • Per-post strategy override - Can be achieved through the ai_contextual_tagging_strategy filter, but an inline UI to do so is possible.
  • Taxonomy toggle settings - Allow users to choose which taxonomies can be auto-generated/suggested (default is just category and post_tag)
  • Confidence Threshold Slider - Allow users to choose the confidence threshold for suggestions

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: TylerB24890 <tyb@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>
Co-authored-by: estelaris <estelaris@git.wordpress.org>
Co-authored-by: karmatosed <karmatosed@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[codecov]

Codecov Report

❌ Patch coverage is 73.69863% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.75%. Comparing base (ad590c8) to head (354fb82).
⚠️ Report is 91 commits behind head on develop.

Files with missing lines	Patch %	Lines
.../Content_Classification/Content_Classification.php	36.19%	67 Missing ⚠️
.../Content_Classification/Content_Classification.php	89.14%	28 Missing ⚠️
...ties/Content_Classification/system-instruction.php	50.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #313      +/-   ##
=============================================
+ Coverage      58.09%   59.75%   +1.65%     
- Complexity       630      701      +71     
=============================================
  Files             46       49       +3     
  Lines           3193     3558     +365     
=============================================
+ Hits            1855     2126     +271     
- Misses          1338     1432      +94     

Flag	Coverage Δ
unit	59.75% <73.69%> (+1.65%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jeffpaul]

1. Should probably center align the button text for Categories and Tags to align to other buttons in the sidebar

2. Can probably remove the loading spinner as the suggest buttons alternate to a loading state already

3. Clicking the suggested term (specifically NOT the X close) appears to also close/remove the suggested term where I thought it might allow me to edit the suggested term (e.g. to pluralize or capitalize it). There doesn't seem to be an obvious way to save suggested terms as the post was already autosaved so there wasn't a Save option in the Gutenberg top toolbar and nothing else obvious in the sidebar panel. What's the expectation to save/apply suggestions?

4. It might be helpful to add a subheading of "Suggested Categories" and "Suggested Tags" just below the separator and just above the suggested terms to make it obvious what that sub-section sidebar panel is?

5. (Continuing from screenshot above) Instead of "Regenerate" perhaps "Suggest again"?

[TylerB24890]

@jeffpaul -- Thanks for the feedback! I've resolved 1, 2 & 4 -- see below regarding 3 & 5;

Clicking the suggested term (specifically NOT the X close) appears to also close/remove the suggested term where I thought it might allow me to edit the suggested term (e.g. to pluralize or capitalize it).

That might be a little challenging with this UI being in the sidebar. We might be better off going with a dedicated modal if we're to support that. Is that something we would want to do as opposed to the sidebar UI?

There doesn't seem to be an obvious way to save suggested terms as the post was already autosaved so there wasn't a Save option in the Gutenberg top toolbar and nothing else obvious in the sidebar panel. What's the expectation to save/apply suggestions?

Clicking the suggestion (outside of the 'X') will automatically create (if new) & assign the post to the term -- but I would have expected the Save button for the post to re-appear, since we changed the post state 🤔 I'll have to see what's going on there.

(Continuing from screenshot above) Instead of "Regenerate" perhaps "Suggest again"?

Personally I feel "Regenerate" makes more sense because the initial button says "Generate", but happy to update it if you prefer.

[dkotter]

Handful of suggestions / comments / concerns, happy to talk through any of those as needed.

Would also be great to get some E2E tests here prior to this getting merged.

[jeffpaul]

That might be a little challenging with this UI being in the sidebar. We might be better off going with a dedicated modal if we're to support that. Is that something we would want to do as opposed to the sidebar UI?

I'm fine holding off for that now as the current implementation is good.

Clicking the suggestion (outside of the 'X') will automatically create (if new) & assign the post to the term -- but I would have expected the Save button for the post to re-appear, since we changed the post state 🤔 I'll have to see what's going on there.

For me clicking the term and not the X also closes/removes the term, but applying/saving it would be better so perhaps something's not working there? I'm running WordPress 7.0-beta5-62069 and AI Provider for Google Version 1.0.3.

(Continuing from screenshot above) Instead of "Regenerate" perhaps "Suggest again"?

Personally I feel "Regenerate" makes more sense because the initial button says "Generate", but happy to update it if you prefer.

Except the initial buttons say Suggest Categories and Suggest Tags?

[dkotter]

Code looks good here to me but in testing, I'm getting the following error:

Bad Request (400) - Invalid schema for response_format 'response_schema': In context=(), 'additionalProperties' is required to be supplied and to be false.

Note I only have OpenAI connected, haven't tested with other Providers

[TylerB24890]

Code looks good here to me but in testing, I'm getting the following error:

Bad Request (400) - Invalid schema for response_format 'response_schema': In context=(), 'additionalProperties' is required to be supplied and to be false.

Note I only have OpenAI connected, haven't tested with other Providers

Must be an OpenAI thing -- it's working with Gemini. I'll debug and push up a fix.

[TylerB24890]

Code looks good here to me but in testing, I'm getting the following error:
Bad Request (400) - Invalid schema for response_format 'response_schema': In context=(), 'additionalProperties' is required to be supplied and to be false.
Note I only have OpenAI connected, haven't tested with other Providers

Must be an OpenAI thing -- it's working with Gemini. I'll debug and push up a fix.

@dkotter I added the additionalProperties key to the schema and OpenAI started working, however Gemini will no longer work until WordPress/ai-provider-for-google#18 is merged.

[dkotter]

Must be an OpenAI thing -- it's working with Gemini. I'll debug and push up a fix.

@dkotter I added the additionalProperties key to the schema and OpenAI started working, however Gemini will no longer work until WordPress/ai-provider-for-google#18 is merged.

Ah, okay, I think that's fine, ideally that PR gets merged and released soon.

Testing again with OpenAI, I am seeing results now but when selecting a term, it's now setting the original taxonomy as the term parent, so I end up with a category term and post_tag term:

In addition, because tags aren't heirarchical, this throws an error anytime I try and save a generated tag, though I only see this error in the console, never reflected in the UI. Fixing the above bug will fix the saving of suggested tags I believe, but if possible, would be ideal to surface any areas so the user knows what's happening. In my case, I clicked on a tag, it disappeared and nothing else happened. I then opened the console and could see the error there.

[TylerB24890]

Must be an OpenAI thing -- it's working with Gemini. I'll debug and push up a fix.

@dkotter I added the additionalProperties key to the schema and OpenAI started working, however Gemini will no longer work until WordPress/ai-provider-for-google#18 is merged.

Ah, okay, I think that's fine, ideally that PR gets merged and released soon.

Testing again with OpenAI, I am seeing results now but when selecting a term, it's now setting the original taxonomy as the term parent, so I end up with a category term and post_tag term:

In addition, because tags aren't heirarchical, this throws an error anytime I try and save a generated tag, though I only see this error in the console, never reflected in the UI. Fixing the above bug will fix the saving of suggested tags I believe, but if possible, would be ideal to surface any areas so the user knows what's happening. In my case, I clicked on a tag, it disappeared and nothing else happened. I then opened the console and could see the error there.

These issues should be resolved. I've added checks to make sure the taxonomy is hierarchical before we attempt to create a parent, and also added checks to ensure the taxonomy isn't returned as the parent.

I've added an error message as well if it fails to create the term, so the user shouldn't be left wondering what happened in that case.

[dkotter]

These issues should be resolved. I've added checks to make sure the taxonomy is hierarchical before we attempt to create a parent, and also added checks to ensure the taxonomy isn't returned as the parent.

I've added an error message as well if it fails to create the term, so the user shouldn't be left wondering what happened in that case.

Just tested and this all looks great now. The only thing I'm seeing is I get the "Changes may not be saved" browser message after generating categories or tags and this shows whether I choose to keep one of those or not.

So if I click Suggest Categories and then after if I click to go back to the post list, I'll get that message. Or if I click Suggest Categories and then click to keep one of the suggestions and then save the post, I'll get that message when trying to go back to the post list screen. I'm assuming there must be some data we're manipulating that shows the post as being "dirty" though I wouldn't expect that, especially in the scenario where I save the post after the generation.

[TylerB24890]

Just tested and this all looks great now. The only thing I'm seeing is I get the "Changes may not be saved" browser message after generating categories or tags and this shows whether I choose to keep one of those or not.

So if I click Suggest Categories and then after if I click to go back to the post list, I'll get that message. Or if I click Suggest Categories and then click to keep one of the suggestions and then save the post, I'll get that message when trying to go back to the post list screen. I'm assuming there must be some data we're manipulating that shows the post as being "dirty" though I wouldn't expect that, especially in the scenario where I save the post after the generation.

Hmm I'm not able to reproduce this. When I click "Suggest Categories" and then click the back button it navigates back without a notice. If I add a suggested category and click Save, then refresh or navigate away, I also do not get a notice 🤔 -- Are there any console logs you can share that might shed some light on what's happening?

[dkotter]

Hmm I'm not able to reproduce this. When I click "Suggest Categories" and then click the back button it navigates back without a notice. If I add a suggested category and click Save, then refresh or navigate away, I also do not get a notice 🤔 -- Are there any console logs you can share that might shed some light on what's happening?

Sorry, false alarm. Tested on a new post and works great. Disabled all AI features and tried to edit original post and getting the problem so something specific about that post is broken (assuming something wrong in the WP beta release). But not an issue with this PR.

Thanks for all the hard work here!