xnode-unit-controller

Xnode Unit Controller This program talks directly with hivelocity to provision Xnode Units / Xnode Ones.

Rewritten due to change in constraints.

How it works roughly:

There's two tables:

1. sponsors
   • Api key for sponsor
   • Initial credit
   • Credit spent
2. deployment
   • Deployment id (Just a number)
   • Nft
   • Sponsor id
   • Provider (Just "hivelocity" for now)
   • Instance id (The deviceId in hivelocity API for reset)
   • Activation date (The date the machine was provisioned, not nft redeem time)

On provision request the server checks if the nft is in the deployment table. If it is, then it will get the API key from the sponsor table (By doing a join on the ID), and reset the machine (Requires shutting down and other stuff unfortunately). If it's not, then it will run the provision request and return the info.

How to use:

1. Make a new postgres database and create the tables using the maketable.sql file as reference.
2. Add login details to env vars, follow .env.sample format.
3. Run go run . to start the server.
4. Add at least one sponsor to the sponsor table:
   1. Set the credits to at least 1000, since each server is about $110 of credit.
   2. Set the API key to an API key you control.

Now you should be able to do: curl -X POST http://localhost:8080/provision/<NFT-ID>

And also: curl -X POST http://localhost:8080/info/<NFT-ID>

For testing, set all the sponsors to the same API key and manually call the endpoint.

Also, there's currently (7-07-2024) an issue with hivelocity where they mark the servers deployed with our product id on their api as "verification" status. So, until that's fixed we can't really test it completely (Since we need the API to return an ip for the system to work). This has now been fixed (30-07-2024).

Testing

1. Set up docker postgres database.
2. Hivelocity tests will require an API key.

TODO

• Parse env vars to get database info.
• Add provision endpoint.
  • Database logic.
  • Add hivelocity provisioning API.
  • Add hivelocity reset API.
    • Shutdown
    • Wait
    • Provision
      • API
  • Test the APIs work.
    • Test reset API works.
    • Set provision api to do round robin over known keys
      • We know that the api for deploy and for provision return the same json anyways, so we should be good.
• Add transactions to avoid race conditions!
• Modify database logic for new tables.
• Retest database logic.
  • Provision new machine.
  • Provision same machine.
• Add cloud-init script to both requests.
• Integrate into DPL.
  • Read parameters from request body.
  • Return expected json.
  • Return errors.
• Add info endpoint.
  • Just parse the hivelocity info endpoint.
  • Why exactly is this needed? I'll integrate the dpl first and then see what's happened.
• Re-enable the provisioning.
  • SAD, hivelocity doesn't always provision servers on our account. Have started a chat with them on slack, but can't do anything until they confirm what's going on.
  • The issue still persists, we're talking to hivelocity for them to find a fix.
• Add mock provisioning.
  • Add an environment variable that "provisions" by resetting a machine from an existing set of machines instead of actually deploying a new vps every time.
  • This would massively simplify testing.
• Estimate cost based on NFT time not just 12 month time.
  • Take NFT activation date (From DPL) and assign enough credit for 12 - months since activation time.
  • Rounded up!
• Robust error handling.
  • API calls should return errors on failure.
  • Check status code of responses.
• Clean up TODOs.
• What does SSL mode do? Might have to enable it for railway?
• Add new table for sponsors
  • Api keys
  • Associate each deployment with a sponsor id
• Associate metadata with each request?
  • Can use tags for this in hivelocity API.
    • Ask hivelocity maximum size for tags in API.
  • Could come in handy if we have to trace machines.
  • Data to store:
    • Xnode UUID
    • Xnode Controller ID
    • Sponsor ID
    • NFT ID
• Add new cloud-init script.
• Chose region based on availability API.
• [o] Testing:
  • Test the system can handle all database events correctly.
  • Test disabling a sponsor doesn't alter it.
  • Test a full deployment works (Up to deploying the machine).
  • Test the environment variables?
  • Test the system can handle all hivelocity API responses gracefully.
  • Test the hivelocity apis are up and working.
  • Test the database can handle 10000 random deployments.
    • Have to mock hivelocity api for this to work.
• Generate table if not provided.
  • Use the sql file?

Maybe worth exploring?

• Generic cloud-init options converter.
  • Turn arguments to proc cmdline things.
• Add more constraints to database to prevent entry errors?
  • Credits should be minimum 1k maybe?
• Improve logs so that they include request information.
  • Which nft is being targetted at least + XnodeId

Before launch

• Might have to enable special account permission on subaccount.
  • "You have not been granted permission to complete this action. Please contact your account manager to grant the proper permission."
  • https://developers.hivelocity.net/reference/post_compute_resource
• Make sure the network is set up correctly:
  • Private networking for xu controller.
  • Private networking for postgres database.
• Make a production version:
  • Make sure the env vars match to the postgres database.
• Make sure it plugs into the DPL
• Ability to manually disable accounts from provisioning new servers. Just add a boolean.

Nice to have

• Add sample env.
• Better error handling
  • Resetting a machine while it still reloading.
  • ALWAYS write provisioned servers to the postgres! Clean up panics to avoid skipping that step.

On Security

• Data (Most importantly sponsor API keys) would ONLY be leaked if:
  • Railway is compromised (anyone with admin access can log into the database and view API keys).
  • There's an SQL injection bug in the xu controller where some user can input a SQL command into some parameter and get the result back somehow.
    • Unlikely because the only "user data" passed from frontend it the nft id, and this is has to be verified on ethereum by the DPL.
    • Also unlikely because the go program does proper SQL parameters, we don't just concatenate strings.
    • ACTUALLY the info request might have this problem, have to parse NFT in DPL in that case.
  • There's a fuck up and the database and/or xu controller aren't isolated from the public internet.
  • There's an exploit in the DPL that lets people send arbitrary get requests to the xu controller.

Notes

• Gock for request mocking? https://medium.com/zus-health/mocking-outbound-http-requests-in-go-youre-probably-doing-it-wrong-60373a38d2aa

• Check maketable.sql for how the schema is created.