There are many pending issues related to the handling of xml name spaces. We need to rethink this. One more try and then we're going to give up. - #7535 encoding/xml: Encoder duplicates namespace tags - #11496 encoding/xml: Serializing XML with namespace prefix - #9775 encoding/xml: Unmarshal does not properly handle NCName in XML namespaces - #12624 encoding/xml: brittle support for matching a namespace by identifier or url - #8167 encoding/xml: disallow attributes named xmlns:* - #11735 encoding/xml: empty namespace conventions are badly documented - #8068 encoding/xml: empty namespace prefix definitions should be illegal - #8535 encoding/xml: failure to handle conflicting tags in different namespaces - #11431 encoding/xml: loss of xmlns= in encoding since Go 1.4 - #7113 encoding/xml: missing nested namespace not handled - #11724 encoding/xml: namespaced and non-namespaced attributes conflict - #7233 encoding/xml: omit parent tags if value is empty #2 - #12406 encoding/xml: support QName values / expose namespace bindings - #9519 encoding/xml: support for XML namespace prefixes

Comment From: aeden

then we're going to give up

Please don't give up :smile:

No doubt that proper namespace handling is a pain, but getting it right is very valuable for those of us who are stuck with legacy APIs that only speak XML.

Comment From: justinlindh-wf

Seconding the comment above. Please don't give up on this. There is still value in handling namespaces properly in golang XML.

Comment From: rsc

I did say we'd try one more time before giving up.

Comment From: glenn-de-backer

+1 I was researching Go as a possible replacement for Python in processing ODT files and I was quickly hitting snags when dealing with multiple namespaces.

Comment From: keltia

I feel your pain, I'm also stuck with SOAP-based web services and the namespace issue is a thorn here. Going to generate XML w/ templates for now. Please do not give up :)

Comment From: rsc

In general we don't pay attention to comments that only say "+1". In fact, I think we have a bot that deletes them.

What would be very useful instead would be pointers to programs (ideally self-contained ones on play.golang.org) illustrating what you are trying to do with encoding/xml, what output you wish you were getting, and how the current output differs/breaks your use case. That information would help guide any eventual fixes we make here.

Thanks.

Comment From: keltia

Point taken. I do not have code right now (because I decided to generate my SOAP requests throught text/template for the moment) but thet fact is, xml.Marshal generate tags w/o any namespace. I tried to use https://github.com/gabstv/go-soap (which use xml.Marshal) and even though there are "xmlns:" declaration for many tags, none of them are in the resulting XML file. Many of the open tickets above are in the same situation.

xml.Unmarshal() is fine.

Comment From: SamWhited

A somewhat specific example I'd like to see work properly without a lot of hacks (I normally pretend the namespace is part of the localname, which works well enough with XMPP because it's such a limited profile of XML, but won't work with more complex documents) is something like the following valid XMPP stream:

<?xml version='1.0'?>
<stream:stream
    from='im.example.com'
    to='juliet@im.example.com'
    version='1.0'
    xml:lang='en'
    xmlns='jabber:client'
    xmlns:stream='http://etherx.jabber.org/streams'>
<stream:error>
  <see-other-host xmlns="urn:ietf:params:xml:ns:xmpp-streams">im.example.net</see-other-host>
</stream:error>
</stream>

which, when decoded and re-encoded will currently result in something like:

<stream
    xmlns="stream"
    from="juliet@im.example.com"
    to="im.example.com"
    version="1.0"
    xmlns:_xml="xml"
    _xml:lang="en"
    xmlns="jabber:client"
    xmlns:_xmlns="xmlns"
    _xmlns:stream="http://etherx.jabber.org/streams">
…

Example here: https://play.golang.org/p/XyMFO301HA

Note that the example reads a raw token because this is XMPP and it has to be streamed which the current XML implementation isn't very good at; that's a separate issue though.

Comment From: pdw-mb

The issue in the above comment is effectively issue #7535

Comment From: ianlancetaylor

See #14407 .

Comment From: klauern

Has there been any work done on this front? I have a big need to work with a bunch of clunky SOAP API's, and these namespaces are just killing me everywhere with it. Are there any libraries that work around this that I could make use of in the interim?

Comment From: rsc

No, there has not.

Comment From: RayfenWindspear

Seems there hasn't been any progress on this. I know it's a dirty workaround, but for Marshalling I ended up just generating a string constant that I just replace with the proper namespaces. Probably just a tad safer than what someone in a duplicate issue did to workaround using text/template

https://play.golang.org/p/MlJEz9PSSO

Comment From: alexrsagen

I work a lot with integration against RFC5730 EPP servers. This highly depends on the encoding/xml package, so i wrote an example showing this use case and how the outlined issues affect it 😄

https://play.golang.org/p/H1s6P95N6tx

I'll be glad to help out with more examples of this use case, just mention me if needed.

Comment From: iwdgo

A short summary of information submitted for the individual issues. Generally speaking, the namespace standard (https://www.w3.org/TR/xml-names) was not correctly enforced. Only #7113 fix is not yet available. No work on extending functionality has been done.

• #7535 Fix submitted • #11496 Commented (invalid syntax) • #9775 Commented (invalid syntax) • #12624 Commented (invalid syntax) • #8167 Improvement request to compact output • #11735 Related to documentation. XMLName usage must be understood by reading Marshal/Unmarshal documentation jointly. • #8068 Fix submitted • #8535 Fix submitted • #11431 No other fix required. Tests have been submitted. (duplicates #8535) • #7113 No fix available yet. • #11724 Fix submitted • #7233 Commented • #12406 Adding func - no work so far • #9519 Improvement request to compact output. (duplicates #8167)

Comment From: iwdgo

7113 has also a proposed fix. All key issues related to the enforcement of namespace standard are solved.

This fix leaves active the handling of the nesting of the tags which would be a basis for other related improvements.

Comment From: iwdgo

All above submitted fixes have minor incompatibilities when merged related to the handling of prefixes. A merged version has been submitted as one. The commit message contains the details previously submitted. Namespace rules are completely enforced. Declaring a namespace is done using xmlns="value w/o prefix" or xmlns:prefix="value with prefix".

The implementation is referring not to eventual prefixes but to their values. To improve XML compatibility, it was using a workaround for this behavior by adding attributes names (using _xmlns) to create chains and consequently almost valid XML. This whole business is now inactive for namespaces as it is not documented. Valid XML is now always returned but sometimes contains oddities as previously.

There are several ways to create namespaces and attributes and various possible conflicts. Further Marshal and Unmarshal share codes so exceptions arise from both types of operations. Idempotency (marshal/unmarshal in cycles w/o changes) is the objective for regular XML and has been mentioned several times in the issues. It is implicit to the documentation. It is not achieved as, for instance, prefix are not restored in tags (cf. TestIssue7535).

The XMLName of a struct must be unique. It designates either the override of the struct name together with an optional non-prefixed namespace. When unmarshaling, the value (the namespace URI usually) is stored into the XMLName of the struct. Without an XMLName field, it is lost.

When marshaling, it can only be an attribute as no prefix is available. A decoder has a strict property which can be false and allows to produce invalid XML. Unmarshal is always strict as it uses Token. Unmarshal can create a prefix only with an attribute.

A Token contains a token.Name.Value used as the value of the namespace when there is one which means that the prefix used is never available if it is not an attribute.

Comment From: alexrsagen

For everyone watching these XML issues in Go and hoping for a fix ASAP, I finally took the plunge and wrote an API-compatible binding to libxml2, which serves great as a temporary workaround for these namespace issues.

https://github.com/alexrsagen/go-libxml - I hope this helps someone!

Comment From: gunnsth

I would like to be able to load XML and write back out similarly, such that the XML is readable. Namespaces are used with a prefix to simplify the XML for readability purposes.

Input XML:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<oval_definitions
    xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5"
    xmlns:ios="http://oval.mitre.org/XMLSchema/oval-definitions-5#ios"
    xmlns:oval="http://oval.mitre.org/XMLSchema/oval-common-5"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://oval.mitre.org/XMLSchema/oval-definitions-5#ios http://oval.mitre.org/language/version5.11/ovaldefinition/complete/ios-definitions-schema.xsd http://oval.mitre.org/XMLSchema/oval-definitions-5 http://oval.mitre.org/language/version5.11/ovaldefinition/complete/oval-definitions-schema.xsd http://oval.mitre.org/XMLSchema/oval-common-5 http://oval.mitre.org/language/version5.11/ovaldefinition/complete/oval-common-schema.xsd">
    <generator>
        <oval:product_name>Creator</oval:product_name>
        <oval:schema_version>5.11.2</oval:schema_version>
        <oval:timestamp>2018-07-11T12:11:47-04:00</oval:timestamp>
    </generator>    
    <tests>
        <ios:global_test check="all" check_existence="at_least_one_exists" id="oval:tst:1" version="1">
            <ios:object object_ref="oval:obj:1"/>
            <ios:state state_ref="oval:ste:1"/>
        </ios:global_test>
    </tests>
</oval_definitions>

When I write this back out with xml.MarshalIndent, the output is

<oval_definitions>
    <generator xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5">
        <product_name>Creator</product_name>
        <schema_version>5.11.2</schema_version>
        <timestamp>2018-07-11T12:11:47-04:00</timestamp>
    </generator>
    <tests>
        <global_test xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5#ios" check="all" check_existence="at_least_one_exists" id="oval:tst:1" comment="" version="1">
            <object object_ref="oval:obj:1"></object>
            <state state_ref="oval:ste:1"></state>
        </global_test>
    </tests>
</oval_definitions>

This makes the XML less readable, defining the xmlns on a tag basis rather than using the prefix/alias closer to the top (top level tag). And with huge XML files that can be ugly to work with.

Example here: https://play.golang.org/p/oDmT9nDhLmu

Comment From: plandem

Any ideas how to fix mix attributes with namespace and without?

<entity id="id" a:id="id_a" b:id="id_b"/>

E.g., this one will work fine:

    type Entity struct {
        IDA         string   `xml:"a id,attr,omitempty"`
        IDB         string   `xml:"b id,attr,omitempty"`
    }

but this one is conflicting:

    type Entity struct {
        ID          string   `xml:"id,attr,omitempty"`
        IDA         string   `xml:"a id,attr,omitempty"`
        IDB         string   `xml:"b id,attr,omitempty"`
    }

Comment From: kumarvaradarajulu

@gunnsth did you figure out a more elegant way to handle the issue.

Comment From: ironytr

i just wanted add empty namespace, used quick fix Xmlns string `xml:"xmlns,attr" but not solving other problems example, int, string types.

tried xml:"* categoryId" turns out -> <categoryId xmlns="*">1001770</categoryId> yet soap api not accepting * character wants <categoryId xmlns="">1001770</categoryId>

type Authentication struct {
    Xmlns string    `xml:"xmlns,attr"  json:"-"`
    AppKey string `xml:"appKey,omitempty" json:"appKey,omitempty"`

    AppSecret string `xml:"appSecret,omitempty" json:"appSecret,omitempty"`
}

type GetSubCategoriesRequest struct {
    XMLName xml.Name `xml:"http://www.n11.com/ws/schemas GetSubCategoriesRequest"`

    Auth *Authentication `xml:"auth,omitempty" json:"auth,omitempty"`

    CategoryId int64 `xml:"* categoryId,omitempty" json:"categoryId,omitempty"` <<<<<<<-------- i need xmlns="" 
}

any help?

Comment From: DemiMarie

@ydnar Can you make a PR?

Comment From: ydnar

@ydnar Can you make a PR?

I did, 3 years ago: https://github.com/golang/go/pull/48641

Comment From: kilianpaquier

Hi any plan to validate and merge the associated PR to fix namespaces soon ?

Comment From: ianlancetaylor

Unfortunately we don't have a plan for how to do that without breaking existing working programs.

Comment From: kilianpaquier

Hi !

Thank you for this reply.

As we don't want to break existing programs, how about introducing encoding/xml/v2 but first go through x/encoding/xml/v2 to ensure no additional fixes or missing features are needed ?

Comment From: ianlancetaylor

@kilianpaquier Yes, I suspect that encoding/xml/v2 is the path forward. But it's a lot of work and I'm not aware of anybody working on it.

Comment From: kilianpaquier

@ydnar Can you make a PR?

I did, 3 years ago: https://github.com/golang/go/pull/48641

Hi @ianlancetaylor, what would be missing from this pull request for a first step ?

It seems to reference and fix all issues linked to this issue ?

Comment From: ianlancetaylor

Experience tells us that any change to encoding/xml that affects existing programs causes some of them to break.

This change (https://go.dev/cl/355353) may be the exception but I don't see why.

Comment From: ydnar

https://go.dev/cl/355353 might be part of a workable solution, it might not.

json/v2 has had a long development process, and JSON is considerably less complex than XML.

I'm hesitant to advocate for (or commit to) a similar approach, even though long-term this probably the right idea.

Maintaining bug-compatibility with v1 via a wrapper seems unworkable for the same reasons encoding/xml hasn't changed.

I've experimented with alternative approaches that could form the basis of an xml/v2 package, but nothing far enough along to propose anything concrete.

@ianlancetaylor how would you think about confining the changes in https://go.dev/cl/355353 behind a GODEBUG flag or go.mod Go version flag?

Comment From: DemiMarie

For encoding/xml/v2, the features I want to see are:

Never accept or produce ill-formed XML.
Never accept or produce namespace-ill-formed XML unless namespace processing has been explicitly disabled.
Reject DTDs by default, as they are a legacy feature with very large attack surface.
Correctly parse and serialize all well-formed XML that does not include a DTD.

Comment From: adamdecaf

There probably needs to be a community lead project that could work on encoding/xml/v2 for a few years to support what is generally needed and to find a new API that could be incorporated in the standard library. IMO nothing is ready to be included in the std lib (or even /x/ repos) right now.

There have been several efforts led by individuals to tackle specific problems with the current stdlib implementation. These should be combined into one project so additional improvements can be included in one module. Not to mention many commonly requested features when using XML in applications (XPath, digital signatures, XSDs, etc)

Comment From: 1e1

Many legacy systems use XML only. Denying Golang stable compatibility with this format is an obstacle to its deployment in these companies. As mentioned above, XML hardly ever evolves. Once compatibility is up to date, maintenance will be light. It's not just a maintenance choice, it's also a strategic one.

Golang encoding/xml: fix name spaces

7113 has also a proposed fix. All key issues related to the enforcement of namespace standard are solved.