I would like to be able to get and display information about tokens per user.

[iocanel]

I am pondering how one should handle the case of multiple users accessing the LLM.

I think that developers should be able to set limits per user. And in order to do that they should be able to obtain information like: tokens per user etc.
This is related to #1407 but not exactly the same.

[geoand]

Can elaborate a little more on this please?

[iocanel]

Let's take the following use case. A team is building a system that gives some AI realted features to the users (e.g. generating images or documents based on some input, tools etc). If the users are not somehow restricted we could have issues, either related to rate limiting, costs etc. So, I think that we should somehow provide this information to developers, which atm seems impossible.

[geoand]

The use case definitely makes sense. I guess what I am struggling with is

So, I think that we should somehow provide this information to developers,

At what point do you envision this happens?

[iocanel]

At what point do you envision this happens?

Not really sure! I guess there are ton of different ways of dealing with this. The simplest way would be to provide an object that could serve as response that could contain at least the info related to the current request.

This is something that users can use as a starting point.

[geoand]

I'm definitely open the idea, just struggling to see how it would be done.

[sberyozkin]

@iocanel This is probably a demo case, where an individual user logs in, and there is some db linking a user email with some in/out token restrictions and the AI service checks that DB via the tool or rag ? Not sure if it is similar at all to what you had in mind, if you think it might be then have a look at say the secure_fraud_detection demo where you can get the current user's email, etc