Can ISO15408 evaluated products be trusted?

[Nora Barrera]

Does anybody understand what is really tested during
an evaluation, or is it just bullshit?





__________________________________
Do you Yahoo!?
Yahoo! Mail - You care about security. So do we.
http://promotions.yahoo.com/new_mail
_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[V***@vt.edu]

Post by Nora Barrera
Does anybody understand what is really tested during
an evaluation, or is it just bullshit?

Ask the vendor for a copy of the evaluation report.

http://csrc.nist.gov/cc/

The *important* part you want to find is the 'Protection Profile' that
it was evaluated against - this replaces the old C1/C2/B1/B2/A levels
in the old DOD Orange Book. Note *very* carefully this change from
Orange Book:

There are *two* components - the Protection Profile (how much stuff the system
is designed to protect) and the EAL (evaluation assurance profile) (how good/
thorough a job the system does). So it's possible to get a very high rating on
a not-very-protective profile (and in fact, many vendors have done this).

http://niap.nist.gov/cc-scheme/pp/index.html has a list of profiles.

Note that the EAL and PP interact - a CAPP (Controlled Access) evaluated at EAL4
may actually provide less *real* protection than an LSPP (Labeled System) evaluated
to EAL3 - the EAL4 just means they've done more work to prove the *provided*
security works as advertised.

The NSA reportedly did an EAL7 light switch. They did a *LOT* of work proving
there was no possible way to subvert any of the security mechanisms the light
switch provided. :)

(And yes, many vendors went for an EAL4 on a lower protection profile instead
of an EAL3 on a profile that required more features - don't let Microsoft, IBM,
Suse, or *anybody* brag up that EAL4 till they tell you what profile it was aginst ;)

[Nora Barrera]

Post by V***@vt.edu
Ask the vendor for a copy of the evaluation report.

But those reports do not contain any valuable
information for me. What kind of tests were done? How?
It looks like security by obscurity.

Post by V***@vt.edu
Note that the EAL and PP interact - a CAPP
(Controlled Access) evaluated at EAL4
may actually provide less *real* protection than an
LSPP (Labeled System) evaluated
to EAL3 - the EAL4 just means they've done more work
to prove the *provided* security works as

advertised.

What's the use of security functions if they can be
circumvented?




Yahoo! Mail
Stay connected, organized, and protected. Take the tour:
http://tour.mail.yahoo.com/mailtour.html

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[HHikita]

Post by Nora Barrera
But those reports do not contain any valuable
information for me. What kind of tests were done? How?

You should look into sections that cover test activity in the CEM.
(5.8, 6.8, 7.9, 8.9)

For EAL4 this would be the following.
8.9.2 Evaluation of Coverage (ATE_COV.2)
8.9.3 Evaluation of Depth (ATE_DPT.1)
8.9.4 Evaluation of Functional tests (ATE_FUN.1)
8.9.5 Evaluation of Independent testing (ATE_IND.2)

The evaluation of the test relys on the developer test documentation
(test plan, test procedure, expected results...which is not disclosed).

Post by Nora Barrera
What's the use of security functions if they can be
circumvented?

It is rather a matter of proving that the Target Of Evaluation (TOE)
adequately protects the Asset.

I would not want to spend US$1,000,000,000,000,000,000 to
protect my wallet (which has about $20 in it right now).

On the other hand I hope the US government spends whatever
is needed to protect the IT system for a nuclear missile
launch system.

So, it should suffice to prove that the security functions
can not be circumvented by an threat agent described in the ST.
__________________________________
Do You Yahoo!?
Upgrade Your Life
http://bb.yahoo.co.jp/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[V***@vt.edu]

What's the use of security functions if they can be circumvented?

Rule #1 of security: It's never perfect.

Rule #2 of security: It's stupid to spend more effort on security than you
need to.

Rule #3 of security: Good security features raise the attacker's cost faster
than they raise your cost. Bad security features are the opposite.

Rule #4 of security: The "right security" is that set of features which raises
the attacker's cost to equal the value of the target, while having the lowest
total sum cost to you.

Almost all bank vaults have security functions (big lockable doors, solid walls,
and so on). The fact that they are still circumventable doesn't mean they're
useless. If the bank has (for instance) an average of $150K in the vault at
a given time, they don't need perfect security - they only need enough security
so it costs an attacker at least $150K to break it. Yes - there's probably some
psycho asshole bank robber who will attack the bank *anyhow*, even if it costs
him $250K and he ends up $100K in the hole. Since it's going to cost you
a lot *more* to stop the $250K attack, your best bet at that point is to
quit improving the security any further, and just shell out the $5K/year in
insurance premiums to cover the bank's losses.. ;) (This also explains why
major branches that may have $3M in cash have lots more sophisticated vaults
than tiny branches, which tend to the wimpier vaults...)

Why is the credit card system basically insecure? Because the banks have
figured out that if they spend $X, the fraud rate will be 3%, but to push it down
to 1% would cost a LOT more $X. What maximizes their return seems to be
spending enough on security to keep the fraud rate around 2%.

Schneier's "Secrets and Lies" has a lot more good stuff to say about this...

[HHikita]

Post by Nora Barrera
Does anybody understand what is really tested during
an evaluation, or is it just bullshit?

For evaluations up to EAL4, the evaluation methods are stated in
"Common Evaluation Methodology"(CEM).
http://www.commoncriteriaportal.org/public/files/cemv2.2.pdf

For evaluations higher than EAL5, the testing method depends on the
country you apply for the certification.

I would trust the ISO15408 evaluated products to do what it states in
its Security Target(ST).
To figure out what is written in the ST is another problem.

__________________________________
Do You Yahoo!?
Upgrade Your Life
http://bb.yahoo.co.jp/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[Nora Barrera]

Post by HHikita
To figure out what is written in the ST is another
problem.

Who understands this strange CC dialect? For me, a ST
is black magic, not a security specification.



Yahoo! Mail
Stay connected, organized, and protected. Take the tour:
http://tour.mail.yahoo.com/mailtour.html

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[HHikita]

Post by Nora Barrera
Who understands this strange CC dialect? For me, a ST
is black magic, not a security specification.

Functional Requirements and Assurance Requirements might seem mind
boggling at first.
But you need a common vocabulary to describe security specifications.
How else would you expect to archive common recognition between all
those countries. :-P

Well actually, for each section of the PP/ST there is a requirement
that says
the section must be coherent. For example CEM says in paragraph 300

"The statement of the TOE description is coherent if the text and
structure of
the statement are understandable by its target audience (i.e. evaluators and
consumers)."

So everything other than those FDP_,FCS_, FIA_, FAU_, ALC_... things,
is supposed to be understandable.

__________________________________
Do You Yahoo!?
Upgrade Your Life
http://bb.yahoo.co.jp/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[Nora Barrera]

Post by HHikita
But you need a common vocabulary to describe
security specifications.

This vocabulary should be understood by more than 100
people.

Post by HHikita
How else would you expect to archive common
recognition between all those countries. :-P

Is this even possible, considering the cultural
differences?
I was told that "internal risk" is not taken into
account in Japan. No employee would hack his own
company.

Post by HHikita
the statement are understandable by its target
audience (i.e. evaluators and consumers)."

How can this be evaluated? The evaluation laboratory
says "Not clear, not understandable". And the guy who
wrote the description answers "you are too stupid to
understand it". What happens next?

Post by HHikita
So everything other than those FDP_,FCS_, FIA_,
FAU_, ALC_... things,
is supposed to be understandable.

_Supposed_
You said it!




Discover Yahoo!
Get on-the-go sports scores, stock quotes, news and more. Check it out!
http://discover.yahoo.com/mobile.html
_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/

[HHikita]

Post by Nora Barrera
I was told that "internal risk" is not taken into
account in Japan. No employee would hack his own
company.

The traditional employment system in Japan was "Shuushin Koyou".
You were basically assured your job until retirement.
So before there were any Information technology, 30years of your
annual income was enough to mitigate most threats.

There are still companies which do not take "internal risk" into
account, and you are able to read about their consequences
in the newspapers daily.

Post by Nora Barrera
How can this be evaluated? The evaluation laboratory
says "Not clear, not understandable". And the guy who
wrote the description answers "you are too stupid to
understand it". What happens next?

The evaluator would at least have to specify where and/or what in the
Security Target
that he finds to be "Not clear, not understandable". And the developer
is given a chance to
take action against these claims.

If the issue is not resolved at the end of the evaluation, then the
verdict would be
"fail" or "inconclusive".

Post by Nora Barrera
_Supposed_
You said it!

You would have to do some homework on the kind of product the PP or ST
is about.

__________________________________
Do You Yahoo!?
Upgrade Your Life
http://bb.yahoo.co.jp/

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/