packages/openai: New OpenAI integration #12494

shmsr · 2025-01-28T08:37:17Z

Proposed commit message

This is a new integration for OpenAI to collect usage metrics using OpenAI's newly launched usage API. It uses the admin key for programmatic access to usage API.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Run tests using elastic-package and manual verification with OpenAI usage dashboard (https://platform.openai.com/usage)

Related issues

Relates x-pack/metricbeat/module/openai: Add new module beats#41516

Screenshots

shmsr · 2025-01-28T12:04:19Z

Update: 28th Jan, 2025

I am still working on the dashboard but rest seems to be more or less done. Except dashboards, other things can be reviewed.

packages/openai/data_stream/audio_speeches/fields/fields.yml

agithomas · 2025-01-29T07:07:32Z

packages/openai/data_stream/audio_speeches/elasticsearch/ingest_pipeline/default.yml

@@ -46,6 +46,10 @@ processors:
        field: openai.audio_speeches.num_model_requests
        target_field: openai.base.num_model_requests
        ignore_missing: true
+    - rename:
+        field: openai.audio_speeches.object
+        target_field: openai.base.object


I am not sure if the word base is a great choice here. Can summary be a better option? I assume the base represents the overall summary of metric within a timeslice.

So, base holds all the common fields. Not summary.

For example, project_id is common across all data_streams.

agithomas · 2025-01-29T07:11:42Z

I refer to the screenshot attached to the issue. Adding a few points for your consideration.

Kindly align the axis label text similar to other gen-ai integration dashboards. It would be best, i think, to have an alignment with the color palette used as well.
When we have a time-series dashboard, please see if there is a need to have "over time" in the panel title.
The need of set underline for the dashboard area splitter (markdown) for texts such as Image Models, Text to Speech Models? Would be good to have a similarity with other gen-ai integration dashboards
For the panels under the sections image models and speech models, kindly correct the y-axis labels
It could be possible in prod environments, the token values can become large. So, for the "single metrics" used at the top of the dashboards, kindly selecting the option "Compact values", if not selected already

shmsr · 2025-01-29T07:20:57Z

@agithomas Yes. Still working on refinements of dashboard. From the issue that Daniela created and also incorporating the comments given by all in AWS Bedrock PR. Will be done soon!

shmsr · 2025-01-29T08:33:53Z

Done!

shmsr · 2025-01-29T08:36:24Z

I refer to the screenshot attached to the issue. Adding a few points for your consideration.

Kindly align the axis label text similar to other gen-ai integration dashboards. It would be best, i think, to have an alignment with the color palette used as well.

When we have a time-series dashboard, please see if there is a need to have "over time" in the panel title.

The need of set underline for the dashboard area splitter (markdown) for texts such as Image Models, Text to Speech Models? Would be good to have a similarity with other gen-ai integration dashboards

For the panels under the sections image models and speech models, kindly correct the y-axis labels

It could be possible in prod environments, the token values can become large. So, for the "single metrics" used at the top of the dashboards, kindly selecting the option "Compact values", if not selected already

Took care of all. For point 3, actually those are clickable links and that's why the underline is there. I didn't put an underline, it is just clickable that takes them to relevant docs.

packages/openai/changelog.yml

packages/openai/_dev/build/docs/README.md

packages/openai/data_stream/code_interpreter_sessions/sample_event.json

packages/openai/data_stream/audio_speeches/sample_event.json

packages/openai/data_stream/audio_speeches/manifest.yml

packages/openai/_dev/build/docs/README.md

muthu-mps · 2025-01-30T08:58:26Z

The panel name Model invocation frequency - by model can be changed to Invocation frequency - by model WDYT?

shmsr · 2025-01-30T09:26:58Z

The panel name Model invocation frequency - by model can be changed to Invocation frequency - by model WDYT?

Yes, makes sense. Thanks!

muthu-mps

LGTM!

ishleenk17 · 2025-01-31T06:27:01Z

@shmsr : I'll be reviewing the code today.
Couple starter comments looking at the dashboard:

Do customers benefit from knowing about API Keys, Project ID's, object types in the dashboard ? Is it worth putting this info there? This info being available in the documents should suffice ?
First 4 panels (invocation frequesncy, token usage....) are generic across models ?
The timestamp is per day, hope that's a discussed value.
We can remove the word "Dashboard" in the title of the dashboard. Just have "OpenAI".

shmsr · 2025-01-31T06:38:37Z

@shmsr : I'll be reviewing the code today. Couple starter comments looking at the dashboard:

Do customers benefit from knowing about API Keys, Project ID's, object types in the dashboard ? Is it worth putting this info there? This info being available in the documents should suffice ?

First 4 panels (invocation frequesncy, token usage....) are generic across models ?

The timestamp is per day, hope that's a discussed value.

We can remove the word "Dashboard" in the title of the dashboard. Just have "OpenAI".

See the screenshot. So, OpenAI usage dashboard does contain top user, projects, api keys (they have mapping of id->name, that's why they can show names; but we have only have ids that they can map or add a processor to add (id->name) mapping in the document and make a visualization out of it) that are using and also it seems a good metric to know who are top n users.

Tokens is not a common concept for all models. Like for image models, no concept of tokens. But it is placed there to maintain consistency across Elastic's LLM integrations. But later, you see the proper seggragation — token usage, image model, audio speeches, etc.
Yes. timestamp is per day wise, because, OpenAI themselves keep the bucket_width of daily. That is data is aggregated over a day; it is configurable though like minute wise, and hour wise. But also the older API used to give daily data. In their dashboard as well, it's daily data. So, kept it like that only.
Sure, on it!

ishleenk17 · 2025-01-31T06:52:55Z

To maintain consistency with other LLM Integrtaions, the agrregated values can be shown for
Total Tokens, Total Invocations. We need not have input output divisioning here in the top panel?
Also we already have them in the Token usage where we are showing input and output.

Do you think its better to have the name as "Overall Invocation Frequency" for the topmost generic panel ?
Since thats the cumulated one. And this is a stacked graph, right ?

ishleenk17 · 2025-01-31T07:03:49Z

packages/openai/data_stream/audio_speeches/agent/stream/cel.yml.hbs

+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/openai-audio-speeches-http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5


For my knowledge, what is this used for ?

For debugging. Suppose there's some issue. So, if you enable this feature, you can see the raw request made by CEL execution and also the response.

< request 1 > < response 1 > < request 2 > < response 2 > ...

Helps in debugging.

ishleenk17 · 2025-01-31T07:06:32Z

packages/openai/data_stream/audio_speeches/agent/stream/cel.yml.hbs

+          "start_time": [string(int(state.initial_start_time))],
+          "page": state.page != null ? [state.page] : [],
+          "bucket_width": ["{{ bucket_width }}"],
+          "group_by": ["project_id,user_id,api_key_id,model"]


Is it like saying if you would have had these fields in the document, these would have become your dimension fields to avoid duplication ?

Let's suppose we consider this example: https://platform.openai.com/docs/api-reference/usage/completions_object

See the response?

Unless you group with project_id,user_id,api_key_id,model, the values for those fields is going to be empty as they are not grouped.

For example this is a real response when dont use group by:

{ "object": "page", "data": [ { "object": "bucket", "start_time": 1738195200, "end_time": 1738281600, "results": [ { "object": "organization.usage.completions.result", "input_tokens": 22, "output_tokens": 124, "num_model_requests": 1, "project_id": null, "user_id": null, "api_key_id": null, "model": null, "batch": null, "input_cached_tokens": 0, "input_audio_tokens": 0, "output_audio_tokens": 0 } ] }, { "object": "bucket", "start_time": 1738281600, "end_time": 1738314727, "results": [] } ], "has_more": false, "next_page": null }

See results? They are nulls.

Yes, they can be dimension fields. But here, we did not enabled TSDB as earlier discussed. Historical data support is there, so we are not enabling the same. Also, we chose logs-* here as discussed earlier.

ishleenk17 · 2025-01-31T07:15:56Z

packages/openai/data_stream/audio_speeches/agent/stream/cel.yml.hbs

+        state.url + "?" + {
+          "start_time": [string(int(state.initial_start_time))],
+          "page": state.page != null ? [state.page] : [],
+          "bucket_width": ["{{ bucket_width }}"],


Say customer changes this aggregation bucket width to say 1m, our dashboards are 1 day timestamp.
Not sure but will that hinder experience for user.
Maybe not, but thoughts ?

So, yes it will be fine. See this screenshot for when bucket width is set to hour 1h. As it x-axis timestamp per day, the viz still properly aggregates. So, it is same as 1d's vizualization.

But yes, if someone set to minute i.e., 1m and suppose collecting 6 months of data, imagine the number of requests.

Calculate per data-stream:

6 months to minutes = 262800. So this how many buckets we have to process.

Now when querying with bucket_width of 1m, OpenAI by default, returns 60 buckets in 1 request. So, 262800/60 = 4380.

So for 8 data_streams, 35040 requests in such short time which will definitely trigger some rate limit. In fact I have tested this as well, it takes a lot of time as OpenAI blocks so much requests in such short time i.e., 35k requests.

I will put the rate limits in another next enhancement as it is not urgent and immediate requirement when going with default or even 1h. But with 1m, there's a huge amount of data to process, so rate limit would make sense there. But then we also expect users to understand that pulling so much historical data with such granularity is definitely a lot.

But yes, giving the users the flexibility to pull data 1d/ 1h / 1m makes sense and that's why I have made it configurable. There might be users who want to make viz and want 1h / 1m of granularity.

alaudazzi

I left a few comments, feel free to address them where appropriate.

packages/openai/_dev/build/docs/README.md

shmsr · 2025-01-31T10:55:58Z

To maintain consistency with other LLM Integrtaions, the agrregated values can be shown for Total Tokens, Total Invocations. We need not have input output divisioning here in the top panel? Also we already have them in the Token usage where we are showing input and output.

Do you think its better to have the name as "Overall Invocation Frequency" for the topmost generic panel ? Since thats the cumulated one. And this is a stacked graph, right ?

@ishleenk17

To maintain consistency with other LLM Integrtaions, the agrregated values can be shown for Total Tokens, Total Invocations. We need not have input output divisioning here in the top panel? Also we already have them in the Token usage where we are showing input and output.

Yes, true. But I put them so that I have space to fill. Because just 2 boxes didn't feel nice. Let me see if I can do something.

Do you think its better to have the name as "Overall Invocation Frequency" for the topmost generic panel ? Since thats the cumulated one. And this is a stacked graph, right ?

Yeah make sense. More clarity.

shmsr · 2025-01-31T11:13:53Z

@ishleenk17 Can you again take a look at the screenshot? I've addressed the comments.

ishleenk17 · 2025-01-31T11:27:44Z

We can have token usage by model. That's what we have done for other LLM's. Would help user to see which model consumes most tokens.

Also, the "i" (info) icons for Total tokens, Totla invocations.

elasticmachine · 2025-02-04T04:12:27Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

AndersonQ · 2025-02-04T06:07:31Z

@shmsr are you sure this PR is correct? It's showing 5000+ files changed. Perhaps it's missing a rebase?

ishleenk17 · 2025-02-04T06:15:55Z

@shmsr are you sure this PR is correct? It's showing 5000+ files changed. Perhaps it's missing a rebase?

@AndersonQ , yes there were some changes across Integrations repo.
I have shared the details on Slack with you.

alaudazzi

LGTM

alaudazzi · 2025-02-04T16:10:29Z

packages/openai/_dev/build/docs/README.md

+
+## Requirements
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.


alaudazzi · 2025-02-04T16:11:54Z

packages/openai/_dev/build/docs/README.md

+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
+
+You need an OpenAI account to access the [Usage API](https://platform.openai.com/docs/api-reference/usage) with a valid [Admin key](https://platform.openai.com/settings/organization/admin-keys) to programmatic access to Usage APIs.
+


@shmsr these two sections -- Requirements and Setup -- should remain separate

elasticmachine · 2025-02-05T08:16:55Z

💚 Build Succeeded

Buildkite Build
Commit: ac36185

History

💚 Build #21599 succeeded 76b768cefcb16bb857e9e7c948460d2378d2b016
💚 Build #21467 succeeded 4416de75ee0e7affd055822b26d57fbf9b9bf6ee
💚 Build #21456 succeeded 8db956170af7b2d47a6c2f2956db96318aec450b
💚 Build #21452 succeeded 631821be6569b970b9c2eb24e44523723be525b0
💚 Build #21443 succeeded 639c6587d4ebc8a7070eb7be345250a730f25218
💚 Build #21432 succeeded 86f1ced2cf3520a189dc9c0f738bb037e60a5cab

cc @shmsr

elastic-sonarqube · 2025-02-05T08:17:04Z

Quality Gate failed

Failed conditions
71.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-02-05T08:34:50Z

Package openai - 0.1.0 containing this change is available at https://epr.elastic.co/package/openai/0.1.0/

shmsr self-assigned this Jan 28, 2025

shmsr added the Integration:openai OpenAI label Jan 28, 2025

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Jan 28, 2025

shmsr requested a review from a team January 28, 2025 12:04

shmsr marked this pull request as ready for review January 28, 2025 22:33

agithomas reviewed Jan 29, 2025

View reviewed changes

packages/openai/data_stream/audio_speeches/fields/fields.yml Outdated Show resolved Hide resolved

agithomas reviewed Jan 29, 2025

View reviewed changes

shmsr mentioned this pull request Jan 29, 2025

packages/openai: Add new OpenAI package #12196

Closed

5 tasks

muthu-mps reviewed Jan 29, 2025

View reviewed changes

shmsr requested review from daniela-elastic and a team January 29, 2025 10:13

daniela-elastic requested a review from alaudazzi January 29, 2025 13:43

daniela-elastic reviewed Jan 29, 2025

View reviewed changes

packages/openai/_dev/build/docs/README.md Outdated Show resolved Hide resolved

daniela-elastic reviewed Jan 29, 2025

View reviewed changes

packages/openai/_dev/build/docs/README.md Outdated Show resolved Hide resolved

muthu-mps approved these changes Jan 30, 2025

View reviewed changes

ishleenk17 reviewed Jan 31, 2025

View reviewed changes

alaudazzi reviewed Jan 31, 2025

View reviewed changes

andrewkroh added Integration:1password 1Password (Partner supported) Integration:abnormal_security Abnormal AI Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Feb 4, 2025

shmsr force-pushed the openai-official-metrics branch from 4416de7 to 76b768c Compare February 4, 2025 11:41

shmsr removed request for a team, AndersonQ and rdner February 4, 2025 11:44

alaudazzi approved these changes Feb 4, 2025

View reviewed changes

qcorporation force-pushed the main branch from 7801e00 to ed31586 Compare February 4, 2025 18:34

packages/openai: New package

ac36185

shmsr force-pushed the openai-official-metrics branch from 76b768c to ac36185 Compare February 5, 2025 08:00

shmsr merged commit fd8c7f8 into elastic:main Feb 5, 2025
4 of 5 checks passed

shmsr mentioned this pull request May 28, 2025

packages/openai: Update dashboard screenshots #14004

Merged

5 tasks


		## Requirements

		You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.

		You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.

		You need an OpenAI account to access the [Usage API](https://platform.openai.com/docs/api-reference/usage) with a valid [Admin key](https://platform.openai.com/settings/organization/admin-keys) to programmatic access to Usage APIs.

packages/openai: New OpenAI integration #12494

packages/openai: New OpenAI integration #12494

Uh oh!

Conversation

shmsr commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

shmsr commented Jan 28, 2025

Uh oh!

Uh oh!

agithomas Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmsr Jan 29, 2025

Choose a reason for hiding this comment

Uh oh!

agithomas commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shmsr commented Jan 29, 2025

Uh oh!

shmsr commented Jan 29, 2025

Uh oh!

shmsr commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Jan 30, 2025

Uh oh!

shmsr commented Jan 30, 2025

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

ishleenk17 commented Jan 31, 2025

Uh oh!

shmsr commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishleenk17 commented Jan 31, 2025

Uh oh!

ishleenk17 Jan 31, 2025

Choose a reason for hiding this comment

Uh oh!

shmsr Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Jan 31, 2025

Choose a reason for hiding this comment

Uh oh!

shmsr Jan 31, 2025

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Jan 31, 2025

Choose a reason for hiding this comment

Uh oh!

shmsr Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmsr Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alaudazzi left a comment

Choose a reason for hiding this comment

Uh oh!

shmsr commented Jan 28, 2025 •

edited

Loading

agithomas Jan 29, 2025 •

edited

Loading

agithomas commented Jan 29, 2025 •

edited

Loading

shmsr commented Jan 29, 2025 •

edited

Loading

shmsr commented Jan 31, 2025 •

edited

Loading

shmsr Jan 31, 2025 •

edited

Loading

shmsr Jan 31, 2025 •

edited

Loading

shmsr Jan 31, 2025 •

edited

Loading

ishleenk17 commented Feb 4, 2025 •

edited

Loading