You don't say what device you're using, so that makes it especially tough to respond. But if you're familiar with Mac Photos, then you know about Smart Albums which make a "collection" based on certain criteria. You can't add or remove pictures from a Smart Album-- if it meets the criteria, then it's there; if not, it isn't.
It makes sense that Trips (and the other sorts of collections) is like that. There's no reason to expect to add or remove pictures.
But you're certainly right that the criteria for Trips albums are totally flaky. I don't know much about Trips, but I just looked at one (on my Mac) in which all the pictures are tagged with the same area on the same day, except for one. Don't know why.
My Nikon gets the GPS location by talking to my phone. But sometimes I apparently don't turn something on fast enough, so a location isn't transmitted. On this trip I have one picture that looks like I just got out of the car and accidentally hit the shutter release, and no location was recorded. But this picture is in that Trips album!
So, apparently, GPS is not needed to get into the Trips album-- but that leaves the question: How does it decide? The answer is, no one knows! I have a Trip that has pictures from Disneyland, San Diego, Georgetown, Texas and Mindenmines, MO -- they are old pictures that I scanned within 10 days of each other. Being scanned, they had no GPS data. But I entered locations for those I knew, and I had to leave many blank. So these pictures aren't connected by location, and barely by date created. But they are a "Trip!"
It's the nature of Artificial Intelligence that no one knows how it works. You give it a bunch of stuff that constitute trips, some stuff that isn't, and let it learn to tell the difference. You don't really know how it's doing it. Perhaps the selection criteria will get better. It'll be interesting to watch.