Dear Martin,
we also had our times when learning the behavior of the Worldmap Panel wrt. InfluxDB. Please try to route your fields to the plugin using an "alias".
While depicted as "boom" in the example below, we usually just call it "metric" ;].
Good luck and with kind regards,
Andreas.
Example


References
While these are coming from a topic related to OpenTSDB (Opentsdb and worldmap panel), it might actually be related and just mean that - while a bit counterintuitive - the ALIAS is the magic button here.
Outlook
We will try to have a look if this can be improved as we are currently working on some improvements to the Worldmap Panel anyway.