[DmitryOlshansky]

Another pull to reduce the amount of operations per every call to advance by shifting them off to the main switch branches. This small set of changes proves to reduce the time to lex by roughly 4%.

Things are getting better and better so we are not likely to see some 10% speed up reached easily with each tweak. BTW where can I get a "breakout" version of dmd that just counts tokens to compare the two (like you did?)

P.S. Would love to see again some bar graphs on the revamped DScanner built with GDC/LDC vs DMD's lexer built with GCC.

Data:
//this pull:
mitry@dmitry-VirtualBox ~/Dscanner $ avgtime -q -r 500 ./dscanner-ldc --tokenCount phobos/std/datetime.d

---

Total time (ms): 12175.5
Repetitions : 500
Sample mode : 24 (224 ocurrences)
Median time : 24.1435
Avg time : 24.3511
Std dev. : 2.04257
Minimum : 22.257
Maximum : 46.198
95% conf.int. : [20.3477, 28.3544] e = 4.00337
99% conf.int. : [19.0897, 29.6124] e = 5.26132
EstimatedAvg95%: [24.172, 24.5301] e = 0.179036
EstimatedAvg99%: [24.1158, 24.5863] e = 0.235293

dmitry@dmitry-VirtualBox ~/Dscanner $ avgtime -q -r 500 ./pull31-dscanner --tokenCount phobos/std/datetime.d

---

Total time (ms): 12879.6
Repetitions : 500
Sample mode : 25 (250 ocurrences)
Median time : 25.35
Avg time : 25.7591
Std dev. : 2.07172
Minimum : 23.628
Maximum : 42.53
95% conf.int. : [21.6986, 29.8197] e = 4.06051
99% conf.int. : [20.4227, 31.0956] e = 5.33641
EstimatedAvg95%: [25.5776, 25.9407] e = 0.181591
EstimatedAvg99%: [25.5205, 25.9978] e = 0.238651