Manage conferences with twilio JS SDK

Question

I want to be able to manage call conferences using a web client (through JS SDK). In the back end I'm using C#, so if that's considered, it'd be better, but at least I want to solve it in general

Scenario:

1. A call comes in
2. A conference is created
3. Agent is called
4. When agent answers through web client, it joins the call
5. Agent should be able to "manage" the conference: put people on hold, add people, etc

I've partially managed to achieve this by returning a TwiML for creating a conference when the first call comes in. Before returning that TwiML I hit the rest API creating a call to the agent and with a URL parameter with the conference ID so the agent joins the conference (using TwiML) once he answers the call.

The issue is that the agent doesn't have the conference ID on the web client side. It just handles it as a normal call. So I can't actually do operations on that conference.

So the question is: how can the agent that joined the conference have the info for the conference to manage it? Is there some library that handles event automatically? When people join, leave, etc?

I want to avoid using database or things like that. I just want to be able to send custom parameters around to handle that using existing twilio JS library. I managed to send and receive custom parameters to client side using TwiML when doing simple calls, but the issue is that when I create the call to the agent using the Rest API on the back end, I have NOT found any way of sending custom/extra parameters