Both protocols have a field indicating the “type” of authentication. There are however differences in the two protocols. In IS-IS, the data associated with the authentication is of variable length. In OSPF it is fixed at 64 bits. 64 bits is sufficient for a password scheme, but would not suffice for a public key signature scheme, which would need a field several hundreds of bits long. A way to circumvent this is by appending a “message” digest at the end of the OSPF packet.
In OSPF there is a single password per link. A router is configured with a password for each link to which it is attached. It transmits that password when it transmits OSPF messages on that link. It expects all OSPF messages it receives on that link to have that password. In IS-IS, a router is configured with a transmit password on a link, which is the password it uses when it transmits IS-IS messages, as well as a set of acceptable receive passwords.
On a P2P link a password scheme in which the receive and transmit passwords are different offers some security. If the passwords are the same, the intruder need only wait for the other router to transmit first, and the intruder will find out the password. Even with two passwords, an intruder can, with effort, discover the passwords.
The reason IS-IS configures routers with a set of acceptable receive passwords, rather than a single receive password, is so that a link, such as a LAN, can be migrated from one password to another without disrupting the network. Since OSPF has single password per link, it is not possible to change the password in an operational network. The routers would all have to be brought down and locally reconfigured.
One of the issues brought by the IS-IS proponents is apparently the big advantage that IS-IS has over OSPF from a security point of view as the former’s packets cannot be routed beyond the immediate next hop or can never be sourced by non-border routers. This can allegedly prevent a variety of potential DoS attacks as anyone can launch OSPF packet bombs in the others network. This apparent vulnerability to DoS attacks is because OSPF rides over IP rather than directly running over the link layer.
Since all OSPF packets can be authenticated using MD5, all spurious OSPF packets can be dropped. But there can be times when MD5 can itself be a part of a problem because it takes significant CPU to check signatures and discard the packets. This is partly true but it is to be noted however that even if OSPF encapsulation is changed to L2, we would still have to support IP encapsulation for virtual links, so we would still have to do MD5. Computing the MD5 used to be a problem in the past when CPUs were not that powerful. However, this is a non issue today, and most of the authentication can be done in the Hardware itself.
Moreover the system administrator can filter on the edges of the network to pry away all the OSPF messages coming from the edges. This will of course be done in addition to cryptography.
There are issues in the HMAC-MD5 scheme as there have recently been reports about attacks on the collision resistance properties of MD5. MD5CRK, was a distributed computing project to break the MD5 hash algorithm in a short period of time. The project closed down with the publication of the paper.
It was discovered that collisions can be found in MD5 algorithm in less than 24 hours, making MD5 very insecure. Further research has verified this result and shown other ways to find collisions in MD5 hashes. We thus need to move away from MD5 towards more complex and difficult to break hash algorithms.
It is because of this that both IS-IS and OSPF now support HMAC-SHA based authentication schemes for the respective protocols. HMAC-SHA support for IS-IS is defined in RFC5310, and for OSPF are been defined in RFC5709.
OSPFv3 has recently been extended to support a similar mechanism as OSPFv2 for authenticating its control packets. The new standard is defined in RFC 6506.