AgilePoint NX form designer can be used even to create really complex forms and does let you even inject your own JQuery into the form using the JavaScript Injector menu to take complete control of the form.
However if you do decide to write your own external HTML5 forms using Angular JS or other modern scripting frameworks which are hosted outside AgilePoint, you can certainly use AgilePoint REST API (JSON format is used) for interacting with process engine. The entire REST API is documented in REST API guide with examples.
So for e.g. if you are looking to kickoff a process, you will be calling “CreateProcInst” API and sample for each of those calls for JavaScript, Java and .net apps is shown in each method’s sample.