sag should be lower (at 3cm maybe, depends on the full travel). Please also measure what the full travel is once you make a hard stop on the front end (but please with warm tires only, don't drop the bike...!). To measure the full, use the zip-tie, too, but when you measure the length, make sure you unload the front completely and measure then from top of the dust cap to the lower of the zip-tie. That is your full travel. The sag when sitting on the bike should not be more than 20% of the full travel without the rider and between 25 - 35% when Rider is on board.
Preloading doesn't make it less soft btw.. To make it more firm, you either need to add some oil (if it is the last inch of the travel) to reduce the air-gap on top (should be between 150 and 170mm, depending on the rider...) or change the oil viscosity (if it is f.e. through the whole range) or you might need different springs (depending on how much you have to pre-load the spring to get to the right sag).
Oil viscosity influences the compression damping, but also the rebound damping on the RGS fork, so a compromise is needed.... On a warmed up fork (by riding it for some miles, the oil starts to warm up and tends to get a bit thinner...), you can make an easy test: stand aside of the bike, engage front-brake and bump the front of the bike down as much as you can. Without letting the handlebar go, give it free play to reflex up again and watch what happens: the suspension should extend to full and go down to the 2-3cm sag you adjusted with the pre-loading of the spring (sag without rider). If it waves more than that, your rebound damping is to low (and so is your compression damping I suppose, as both sides work the same on the RGS). In that case, you need to go up with the oil viscosity. Personally, I only buy 5 grade and 10 grad and mix it until it works... if it does not extend beyond the adjusted sag-point, your oil-viscosisty grade is to high and you need lighter oil (f.e. if you are at 10, you need to go down to 7,5 etc.).
you can take some nice infos out of the internet, f.e.: http://www.gostar-racing.com/club/motorcycle_suspension_set-up.htm#Setting your bike up
though this talks more about full adjustable suspension, you can also see he has a section for when the suspension is not fully adjustable. In that case, your oil viscosity as said is how you set the way the suspension travels.