The city's decision to invalidate the results of a firefighter exam after discovering a sample test existed was shocking, according to the CEO of the company that sold the test materials to the city.

"It is common practice in the testing industry to make preparation materials available to test takers," Fred Rafilson, head of I/O Solutions, Inc. wrote in an email response to the city's announcement that it would invalidate the test.

Ken Nakamatsu, city director of Human Resources, and Fire Chief Attilio Leonardi announced Thursday that a test taken by more than 2,000 would-be Hono-lulu firefighters would be invalidated after the city discovered some candidates had found and purchased a practice test offered by I/O Solutions.

The practice test was very similar to the actual test in overall structure and in the structure of the individual questions, city officials said, and because not all the firefighter candidates were aware the test existed, those who found it on I/O Solutions Web site had an unfair advantage.

Nakamatsu said the city was not aware the practice test was available before it administered the exam.

But Rafilson wrote that the practice test was appropriate, fairly constructed and the city was aware of its existence before it purchased the test.

Nakamatsu said again yesterday that the city was not aware the practice test was available until after the actual test was administered to the candidates.

Practice tests are made available for nearly all major academic, job certification and licensure examinations, Rafilson wrote.

"In fact," he wrote, "it is considered unfair when candidates do not have materials available to them because some candidates have less experience taking tests than others and may have high levels of test anxiety, which can artificially lower their scores.

"In other words," he wrote, "preparation materials increase the overall reliability and fairness of the examination."

Thousand of public service agencies that do business with his company require that preparation materials be made available to candidates, he said.

"Please note that not a single item in any of our test materials has ever appeared on an actual exam," he wrote. "In addition, the city was aware of all our offerings before making the decision to use our exam."

"Needless to say," Rafilson wrote, "we are shocked by this decision."

Nakamatsu said yesterday, "We'll stand by our statement that, in consulting with independent test experts, we still believe that some candidates may have an unfair advantage on the fire test, and as a result we stick by our decision to cancel."

The independent test experts included a testing specialist from the state Department of Education and the head of the recruitment and examination division of the judiciary, he said. Those two people concurred with the decision made by six testing specialists from his office, he said.

In late October, the city canceled a video test on the day it was to be administered to firefighter candidates for similar reasons, Nakamatsu said.

Ergometrics of Edmonds, Wash., the company that sold the city the video test, also offered a practice test the city considered an unfair advantage to those who discovered it on the Ergometrics Web site, he said.

That company refunded the city's testing costs of $8,000, Nakamatsu said.

He said I/O Solutions is being asked to refund the $23,500 the city paid for its tests.

Rafilson could not be reached for comment on the refund yesterday, and had not addressed the subject in his e-mail.

City officials said the 2,309 candidates who took the test on Sept. 10 will receive letters instructing them to report for a new test on Feb. 25.

Neighbor Island and Mainland candidates provided their own transportation to the O'ahu testing sites for the first tests and will be required to do the same for the second test, Nakamatsu said.

Reach Karen Blakeman at kblakeman@honoluluadvertiser.com.