Actually I DID IT! YAY!
So StartsAt is an object, and you can actually call an arbitrary object methos in go template. And it happens Time object has Add and Sub methods, and duration is in milliseconds.
{{- .Annotations.URL -}}&from={{- (.StartsAt.Sub -3600000000000).UnixMilli -}}&to={{{- (.StartsAt.Add 300000000000).UnixMilli -}}