Base64 error?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Base64 error?

Colm O hEigeartaigh-2
Hi,

I'm doing some testing with various BASE-64 implementations and I think there's an error in the Santuario Base64 encoder to do with whitespace. If so though it's been there a looong time without anyone noticing...

The BASE64 implementation is here:

https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java

In the "encode" method it's adding a newline character with:

encodedData[encodedIndex++] = 0xa;

However this is just "\n". The RFC defines a CRLF as "\r\n":

https://www.ietf.org/rfc/rfc2045.txt

It looks like a bug...but would like some feedback from others more familiar with the RFC.

Colm.


--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

Alessio Soldano
I wonder which implications this could have in terms of
interoperability... ?

Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:

> Hi,
>
> I'm doing some testing with various BASE-64 implementations and I
> think there's an error in the Santuario Base64 encoder to do with
> whitespace. If so though it's been there a looong time without anyone
> noticing...
>
> The BASE64 implementation is here:
>
> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
>
> In the "encode" method it's adding a newline character with:
>
> encodedData[encodedIndex++] = 0xa;
>
> However this is just "\n". The RFC defines a CRLF as "\r\n":
>
> https://www.ietf.org/rfc/rfc2045.txt
>
> It looks like a bug...but would like some feedback from others more
> familiar with the RFC.
>
> Colm.
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com


--
Alessio Soldano
Web Service Lead, JBoss

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

Colm O hEigeartaigh-2
I did some interop testing with the Commons Codec Base64 implementation and the JDK8 java.util one, and the output is different. I have to explicitly use new byte[] {'\n'} for the line break to get them to work with Santuario.

Colm.

On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <[hidden email]> wrote:
I wonder which implications this could have in terms of interoperability... ?


Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
Hi,

I'm doing some testing with various BASE-64 implementations and I think there's an error in the Santuario Base64 encoder to do with whitespace. If so though it's been there a looong time without anyone noticing...

The BASE64 implementation is here:

https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java

In the "encode" method it's adding a newline character with:

encodedData[encodedIndex++] = 0xa;

However this is just "\n". The RFC defines a CRLF as "\r\n":

https://www.ietf.org/rfc/rfc2045.txt

It looks like a bug...but would like some feedback from others more familiar with the RFC.

Colm.


--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com


--
Alessio Soldano
Web Service Lead, JBoss




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Base64 error?

Cantor, Scott
> I did some interop testing with the Commons Codec Base64 implementation
> and the JDK8 java.util one, and the output is different. I have to explicitly use
> new byte[] {'\n'} for the line break to get them to work with Santuario.

Could you define "work with"?

I know we use commons in our code and I don't think we've had issues consuming signatures. If you mean "get them to produce the same encoded output", that's understandable.

-- Scott

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

Marc Giger
In reply to this post by Colm O hEigeartaigh-2
Hi Colm,

XMLDsig and XMLEnc referring RFC 2045 (Base64 MIME) as you
already noted. After reading many sources I also come to
the conclusion that

a) Line breaks MUST be inserted after 76 characters
b) Line breaks MUST be CRLF sequences

The Java-Doc of java.util.Base64 and
org.apache.commons.codec.binary.Base64
also confirms that statement.

To get the same output from both implementations you
can/should use
java.util.Base64.getMimeEncoder() // RFC2045
and
org.apache.commons.codec.binary.Base64.Base64(final boolean
urlSafe) // RFC2045

or other constructors while specifying explicitly the line lengths and
newline characters.


In the case of commons codec the _decoder_ should work with none/LF/CRLF
because it simply ignores unexpected values.
I can imagine that other decoder impl. do the same. Santuario
should most probably do the same for max. interoperability.

Btw, it looks like I got it also wrong (because I didn't read the RFC)
in the StAX impl. If it happens that you fix the Base64 impl. could
you also correct the new Base64OutputStream(...) call in
AbstractEncryptOutputProcessor ?

Thanks and happy coding,

Marc




On Mon, 30 May 2016 14:32:30 +0100
Colm O hEigeartaigh <[hidden email]> wrote:

> I did some interop testing with the Commons Codec Base64
> implementation and the JDK8 java.util one, and the output is
> different. I have to explicitly use new byte[] {'\n'} for the line
> break to get them to work with Santuario.
>
> Colm.
>
> On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <[hidden email]>
> wrote:
>
> > I wonder which implications this could have in terms of
> > interoperability... ?
> >
> >
> > Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
> >  
> >> Hi,
> >>
> >> I'm doing some testing with various BASE-64 implementations and I
> >> think there's an error in the Santuario Base64 encoder to do with
> >> whitespace. If so though it's been there a looong time without
> >> anyone noticing...
> >>
> >> The BASE64 implementation is here:
> >>
> >>
> >> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
> >>
> >> In the "encode" method it's adding a newline character with:
> >>
> >> encodedData[encodedIndex++] = 0xa;
> >>
> >> However this is just "\n". The RFC defines a CRLF as "\r\n":
> >>
> >> https://www.ietf.org/rfc/rfc2045.txt
> >>
> >> It looks like a bug...but would like some feedback from others more
> >> familiar with the RFC.
> >>
> >> Colm.
> >>
> >>
> >> --
> >> Colm O hEigeartaigh
> >>
> >> Talend Community Coder
> >> http://coders.talend.com
> >>  
> >
> >
> > --
> > Alessio Soldano
> > Web Service Lead, JBoss
> >
> >  
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Base64 error?

Cantor, Scott
In reply to this post by Colm O hEigeartaigh-2
> However this is just "\n". The RFC defines a CRLF as "\r\n":
>
> https://www.ietf.org/rfc/rfc2045.txt
>
> It looks like a bug...but would like some feedback from others more familiar
> with the RFC.

The RFC seems pretty clear, and unlike some specs, XML Sig is pretty explicit about referencing it.

The C++ library delegates base64 to the cryptographic libraries, and I don't know offhand what OpenSSL does by default.

Most base64 decoders are very liberal about accepting anything re: whitespace, obviously. But if you're going to fix this, I would allow some time for people to test it.

-- Scott

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Base64 error?

Clement_Pellerin
> However this is just "\n". The RFC defines a CRLF as "\r\n":

The XML spec requires the XML processor to map \r\n to \n on input,
thereby hiding this discrepancy.

Is the Base64 value ever used without this mapping taking place?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Base64 error?

Cantor, Scott
> The XML spec requires the XML processor to map \r\n to \n on input,
> thereby hiding this discrepancy.

Good point.

> Is the Base64 value ever used without this mapping taking place?

Possibly in very obscure cases involving enveloping signatures?

-- Scott

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

DSosnoski
In reply to this post by Marc Giger
I'd thought that the line length restriction had been overridden
somewhere for XML Sig, but I don't see where that's specified. There was
a discussion of this whole issue with the intent to change it:
https://lists.w3.org/Archives/Public/w3c-ietf-xmldsig/2001AprJun/0195.html

XML Schema requires processors to ignore the line-length limitation and
collapse whitespace sequences to single space characters, so in schema
terms this all becomes irrelevant, and XML Sig says the schema is
normative. But it also says MIME is normative. Which is more normative
than the other?

   - Dennis

On 05/31/2016 05:49 AM, Marc Giger wrote:

> Hi Colm,
>
> XMLDsig and XMLEnc referring RFC 2045 (Base64 MIME) as you
> already noted. After reading many sources I also come to
> the conclusion that
>
> a) Line breaks MUST be inserted after 76 characters
> b) Line breaks MUST be CRLF sequences
>
> The Java-Doc of java.util.Base64 and
> org.apache.commons.codec.binary.Base64
> also confirms that statement.
>
> To get the same output from both implementations you
> can/should use
> java.util.Base64.getMimeEncoder() // RFC2045
> and
> org.apache.commons.codec.binary.Base64.Base64(final boolean
> urlSafe) // RFC2045
>
> or other constructors while specifying explicitly the line lengths and
> newline characters.
>
>
> In the case of commons codec the _decoder_ should work with none/LF/CRLF
> because it simply ignores unexpected values.
> I can imagine that other decoder impl. do the same. Santuario
> should most probably do the same for max. interoperability.
>
> Btw, it looks like I got it also wrong (because I didn't read the RFC)
> in the StAX impl. If it happens that you fix the Base64 impl. could
> you also correct the new Base64OutputStream(...) call in
> AbstractEncryptOutputProcessor ?
>
> Thanks and happy coding,
>
> Marc
>
>
>
>
> On Mon, 30 May 2016 14:32:30 +0100
> Colm O hEigeartaigh <[hidden email]> wrote:
>
>> I did some interop testing with the Commons Codec Base64
>> implementation and the JDK8 java.util one, and the output is
>> different. I have to explicitly use new byte[] {'\n'} for the line
>> break to get them to work with Santuario.
>>
>> Colm.
>>
>> On Mon, May 30, 2016 at 1:49 PM, Alessio Soldano <[hidden email]>
>> wrote:
>>
>>> I wonder which implications this could have in terms of
>>> interoperability... ?
>>>
>>>
>>> Il 30/05/2016 12:30, Colm O hEigeartaigh ha scritto:
>>>  
>>>> Hi,
>>>>
>>>> I'm doing some testing with various BASE-64 implementations and I
>>>> think there's an error in the Santuario Base64 encoder to do with
>>>> whitespace. If so though it's been there a looong time without
>>>> anyone noticing...
>>>>
>>>> The BASE64 implementation is here:
>>>>
>>>>
>>>> https://svn.apache.org/repos/asf/santuario/xml-security-java/trunk/src/main/java/org/apache/xml/security/utils/Base64.java
>>>>
>>>> In the "encode" method it's adding a newline character with:
>>>>
>>>> encodedData[encodedIndex++] = 0xa;
>>>>
>>>> However this is just "\n". The RFC defines a CRLF as "\r\n":
>>>>
>>>> https://www.ietf.org/rfc/rfc2045.txt
>>>>
>>>> It looks like a bug...but would like some feedback from others more
>>>> familiar with the RFC.
>>>>
>>>> Colm.
>>>>
>>>>
>>>> --
>>>> Colm O hEigeartaigh
>>>>
>>>> Talend Community Coder
>>>> http://coders.talend.com
>>>>  
>>>
>>> --
>>> Alessio Soldano
>>> Web Service Lead, JBoss
>>>
>>>  
>>
>> --
>> Colm O hEigeartaigh
>>
>> Talend Community Coder
>> http://coders.talend.com
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

Colm O hEigeartaigh-2
In reply to this post by Cantor, Scott
Thanks all for the feedback. I came across this bug in Apache Ranger, which uses the Santuario BASE-64 encoder to encode keys as part of a Hadoop Key Management Service. I swapped the Santuario BASE-64 encoder out and replaced it with Commons Codec, and the output is different. To produce the same output, I need to do something like:

      String masterKey1 = org.apache.xml.security.utils.Base64.encode(key.getBytes());
      org.apache.commons.codec.binary.Base64 base64 = new org.apache.commons.codec.binary.Base64(76, new byte[]{'\n'});
      String masterKey2 = new String(base64.encode(key.getBytes())).trim();
      org.junit.Assert.assertEquals(masterKey1, masterKey2);

There is actually two problems here, using '\n' instead of '\r\n' and also not adding an additional line separator at the end (hence the trim() above on the commons codec output).

Given, as Clemont points out, that the XML processor maps \r\n to \n on input, I'm inclined to make the change as it won't have huge backwards compatibility implications. Anyone object to this?

Colm.

On Mon, May 30, 2016 at 9:09 PM, Cantor, Scott <[hidden email]> wrote:
> The XML spec requires the XML processor to map \r\n to \n on input,
> thereby hiding this discrepancy.

Good point.

> Is the Base64 value ever used without this mapping taking place?

Possibly in very obscure cases involving enveloping signatures?

-- Scott




--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Base64 error?

Cantor, Scott
In reply to this post by DSosnoski
> XML Schema requires processors to ignore the line-length limitation and
> collapse whitespace sequences to single space characters, so in schema
> terms this all becomes irrelevant, and XML Sig says the schema is
> normative. But it also says MIME is normative. Which is more normative
> than the other?

XML Schema is not a presumed part of the XML Signature processing model, and in fact XML Signature fails badly in a number of scenarios involving schema validation, so it is generally bad practice to combine them without a lot of extra care.

-- Scott


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Base64 error?

Colm O hEigeartaigh-2

On Tue, May 31, 2016 at 3:02 PM, Cantor, Scott <[hidden email]> wrote:
> XML Schema requires processors to ignore the line-length limitation and
> collapse whitespace sequences to single space characters, so in schema
> terms this all becomes irrelevant, and XML Sig says the schema is
> normative. But it also says MIME is normative. Which is more normative
> than the other?

XML Schema is not a presumed part of the XML Signature processing model, and in fact XML Signature fails badly in a number of scenarios involving schema validation, so it is generally bad practice to combine them without a lot of extra care.

-- Scott





--
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com
Loading...