In a work context, “We’re all going to lunch” usually carries the implication of “Would you like to join us?”
If you’re still not sure if the implication is present, you can ask “Mind if I come along?” Or “Do you have space for one more?” Or something similar.
Here is an Instagram video about this!
