Delivery Service Active Flag Rework

[ocket8888]

In accordance with the blueprint from #4654, this changes the active property of Delivery Services from a simple boolean to a three state system, where the three states are:

• ACTIVE - The DS is routed and cache servers will generate configuration for it. The equivalent of true today.
• PRIMED - The DS is not routed, but cache servers will still generate configuration for it. The equivalent of false today.
• INACTIVE - The DS is not routed and cache servers will not generate configuration for it. Has no equivalent today.

Additionally, because I was making a new DS model already anyway since that's a breaking change, I made some things that are never allowed to be null or undefined into non-reference values (e.g. XMLID is a string not a *string etc.) and made lastUpdated format in RFC3339 in JSON responses.

The new model now totally drops LongDesc1 and LongDesc2, which were deprecated in the now-itself-deprecated APIv3 and unavailable for direct use in APIv4 - as an interesting consequence of this and how DSRs are stored internally, it is no longer possible to make a DSR that will set a DS's longDesc1 and/or longDesc2 to anything but null even using APIv3. They are shown as null in responses, but are otherwise totally ignored by Traffic Ops. DS longDesc1 and longDesc2 fields can still be manipulated directly and viewed as normal in APIv3.

I also made some changes to the testing utilities, because they weren't registering themselves as helpers so it was difficult to find actual failures.

Fixes #3746

This PR includes documentation, TO changes, and database migrations, but does not actually make the new state work in t3c, nor does it update Traffic Portal with the ability to deal with it. These efforts are left to a later date.

---

Which Traffic Control components are affected by this PR?

• Documentation
• Traffic Ops Client (Go)
• Traffic Ops

What is the best way to verify this PR?

I've updated the tests, and really there's only one new DS permutation, which has a test case added, so there's not really a whole lot more to do than just run the tests. You can replicate them by using APIv5 to manipulate DSes and DSRs if you like, just make sure everything looks okay.

If this is a bugfix, which Traffic Control versions contained the bug?

• 7.x
• 6.x

PR submission checklist

• This PR has tests
• This PR has documentation
• This PR has a CHANGELOG.md entry
• This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY

[rimashah25]

Tested the API tests locally, all work (v3/v4/v5) fine. Yet to test DS, DSRs creation with API v5 curl cmds.

[ocket8888]

This PR depends on #7156

[ocket8888]

I need to change the migration file names because new migrations have been added since this PR opened. I'm in a meeting atm, I'll do it this afternoon.

[rimashah25]

Tested and reviewed the following:

1. Able to create DS with 3 states (Active, Primed, Inactive) using APIv5
2. Able to create DS with 2 states (as expected) (Active=true, Primed=false) using APIv3, APIv4, APIv4.1
3. See the timestamp format update in APIv5 ONLY
4. Could create DS and DSR with longDesc1 and longDesc2 in the JSON payload with all 3 API versions. Per the description in the PR, I shouldn't be able to do so.
5. Routing Name can be changed to default #7094 didn't seem like it was fixed. I am able to change the routingName even when the value is set to cdn
6. GET receives the right DS structure (3.0 and 4.0 doesn't show regional field but 4.1 and 5.0 shows it)
7. Update to testing utilities makes sense to implement.
8. Variable name changes are tested in all three versions of API tests
9. DB migration files need timestamp change.
10. Also tested the following APIs with v3.0, 4.0, 5.0 and they show the right DS structure: /cdns/{{CDN Name}}/configs/monitoring, /cdns/{{CDN Name}}/snapshot/new, /deliveryservices/{{ID}}/safe, /servers/{{ID}}/deliveryservices, /snapshot
11. /users/{{ID}}/deliveryservices cannot curl this API with any versions

[ocket8888]

It looks like I accidentally removed regional from the docs when I rebased them after it was added, so I'll fix that and the migration timestamps. Other than that:

4. Could create DS and DSR with longDesc1 and longDesc2 in the JSON payload with all 3 API versions. Per the description in the PR, I shouldn't be able to do so.

I think maybe I should clarify. You should be able to create a DS and DSR with any extra fields specified that you want. You could set "foo": "bar" and it shouldn't break anything. But it just shouldn't be stored. The same thing should happen with longDesc1 and longDesc2 in APIv5. Requests containing those should succeed, but no matter what you set them to, they should be null when you GET it in APIv3. Also, because DSRs are always stored at the latest API version, you should be able to create a DSR that sets a DS's longDesc1/longDesc2 to anything you want, but even if you do that in APIv3 it should be null when you GET it afterward.

Also, the legacy structures for DSes use the omitempty directive in the json portion of their struct tags for those fields, so when longDesc1/longDesc2 is/are NULL in the database, it/they are omitted from the response, so in that case it looks the same as if I had removed them from that version too (but I didn't I swear). Basically those will always be missing from DSRs in APIv3 but that's the same as if you used that version to set them explicitly to null before these changes.

That said, it looks like DSRs are always showing APIv5 representations in responses to POST requests on this branch - which doesn't fail tests because I guess the APIv3 client doesn't decode/return the response to those requests. Which I can't add because it'd be a breaking change, so I'll just need to fix and test manually, since a test for that can't really be written easily.

5. [Routing Name can be changed to default #7094](https://github.com/apache/trafficcontrol/issues/7094) didn't seem like it was fixed. I am able to change the routingName even when the value is set to cdn

That issue doesn't really exist. I think when I checked for it on master I forgot to generate SSL Keys for the delivery service first. I'll close that.

11. `/users/{{ID}}/deliveryservices` cannot curl this API with any versions

This endpoint existed in API versions 1 and 2, which existed when that blueprint was written but no longer do. So it's fine to be unreachable, since it would've been before anyway.

[rimashah25]

Today, if I give longDesc1 and longDesc2 in 4.0 this is the error I see:

curl -k -X POST -H "Content-Type: application/json" --cookie "mojolicious=eyJhdXRoX2RhdGEiOiJhZG1pbiIsImV4cGlyZXMiOjE2Njc1OTc4MjIsImJ5IjoidHJhZmZpY2NvbnRyb2wtZ28tdG9jb29raWUifQ--f85d59e0865cbcf9bac01d3fc42b5e12e7c8f8bf" --data @ds_v4.json "https://localhost:8443/api/4.0/deliveryservices"
{"alerts":[{"text":"the longDesc1 and longDesc2 fields are no longer supported in API 4.0 onwards","level":"error"

So, why is the same strategy not followed for 4.0 and 5.0 in this PR?

[rimashah25]

A few minor comments but overall the PR LGTM from functionality and integration tests perspective.

[zrhoffman]

Looks good.