Hi, AFAIK there is a 0ohm resistor on the module (near the µFL connector) to remove if you want external ANT/GND connection.
No need to set in software, I think you have to design a trace with 50ohm impedance and solid ground plane from the ANT pin.
I make this on my pcb with AM02 module, the 50ohm impedance trace go to the positive pin of a SMA board edge footprint. For now I solder a little helical 868MHz antenna and I have good performance.
(not a specialist but I try to improve my designs)
See my post http://community.heltec.cn/t/am02-module-plus-custom-pcb-minimal-circuit-arduino-compatible/3862/11?u=alexfr and don’t hesitate to share your design choices. (especially on the power management/ADC battery or ADC readings).