I would suggest going to Fraser - Winter Park instead of Granby. If everything is running on-time, you will have nearly six hours to kill. Depending on how #6 is running, it could be a whole lot more. There is nothing, and I mean absolutely nothing to do in Granby in the winter. I rode through Granby last Friday on #5 and there was no one off and no one on. We stopped, sat for ten seconds, and off we went. In the summer Granby is the jumping off point for Rocky Mountain National Park and some local tourist ranches, but in the winter there is nothing. I'm not sure how you would spend six hours (or more) there without going crazy.
The Fraser stop, 13 miles before Granby, allows you to get to Winter Park ski area where there is lots to do to occupy your time. You would lose the portion of the trip through the Fraser River Canyon, but otherwise you would still see all of the route up to and through the Moffat Tunnel.