I personally wouldn't like that, because I'm a very private person and I want my yard to be my yard only. I would much rather prefer just an arranged playtime where the dog comes over for a couple of hours, then is picked up or taken back home.
Would there be a way to lock and open the gate from either side? Do you think there would be a disturbance is your neighbor wanted the gate open, and for any reason, you wanted it closed to be alone with your dog in the yard or inside? Would it create bad feeling between you and the neighbor?
These are just hypothetical situations of possible downsides.
What if your dog didn't want to come back through the gate into your yard, say for the night or for anytime you wanted the gate close and be with him alone? Would you then have to force him through the gate and have him resent you for taking him away from his buddy? What if the other dog didn't want to go back home, say at night?
What if your dog is in their yard and they have company. Their garage or front door is left open by one of their visitors and he takes off.
If the gate is big enough for the dogs to go through, it's that much easier for a person on their side of the fence, maybe a visitor, to enter your yard with no effort.
I think your dog would bond with you better and feel that he really has his own home and yard to enjoy or protect, as opposed to a community yard.
What if you both decide to close the gate, and the dogs are so used to it already that they start digging under it or clawing at it to get it open so they can be with each other. Once they're introduced to that arrangement, they may object to it being closed.
There's a saying that fences make good neighbors, this isn't a total fence, just a gate, but it is an opening nevertheless.
In my opinion, Hunter needs to feel secure in his own home and in his own yard with his mom. Visiting can be done by mutual agreement at any time, and then the dogs go back to their respective homes/yards. The yard will lead to time inside your house, not realistic to think it won't. You can't go inside with Hunter for any reason without the neighbor's dog outside in your yard making a fuss, or your dog on the inside whining to get out to his friend.
Many years ago we had a neighbor that said if we ever wanted to go on vacation for a few weeks, he would just take off a few planks of our fence and let him go to their house like that, or their dogs in our yard to be with him. I thanked him for the offer, but never took him up on it, too weird for my comfort.
If I were you I'd really think about this before agreeing to it.