Hi Furzedo
The best thing is to use the WooCommerce order no as the reference in Xero and not as the invoice no. Generating invoices in Woo as well as Xero means that you will have a lot of problems keeping them in synch eg if you take a telephone order or a physical shop order directly into Xero it will not show in Woo and immediately sets the two systems out of synch. It will also be confusing to the customer to get two invoices for the same thing, which might not even match for a whole variety of reasons!
So decide which system you want to use to invoice the customer and the order ref acts as a link to Xero to reconcile the two systems. If it was me I would use Xero for invoicing and use the Woo plugin for the GRN/packing slips only especially as there is often some adjustments that need making manually in Xero.
Thanks
Pete