Array types creates extra (and unneeded) type for the array items

[tlandschoff-scale]

Deriving Array(DemoType) from a given type DemoType creates an unneeded DemoTypeArray_DemoTypeType for the items of the array.

I created a simple test for this and used git bisect to identify the commit which caused this change in behaviour. The change was in e162935.

Here is the code to demonstrate this:

#!/usr/bin/env python

try:
    import logging
    import uuid

    from lxml import etree

    from spyne.application import Application
    from spyne.decorator import srpc
    from spyne.interface import Wsdl11
    from spyne.protocol.soap import Soap11
    from spyne.service import ServiceBase
    from spyne.model.primitive import Unicode
    from spyne.model.complex import Array

    target_namespace = "target.namespace"

    DemoType = Unicode(
        type_name="DemoType",
        pattern="demo",
        __namespace__=target_namespace,
    )

    class ServiceWithNewType(ServiceBase):
        @srpc(Array(DemoType), _returns=DemoType)
        def derive_uuid(original):
            return []


    logging.basicConfig()
    app = Application([ServiceWithNewType], target_namespace, in_protocol=Soap11(), out_protocol=Soap11())
    app.transport = 'null.spyne'

    wsdl = Wsdl11(app.interface)
    wsdl.build_interface_document('URL')


    wsdl_tree = etree.fromstring(wsdl.get_interface_document())

    defns = etree.Element("matches", nsmap={"xs": "http://www.w3.org/2001/XMLSchema"})

    for defn in wsdl_tree.xpath(
            "//xs:schema[@targetNamespace='target.namespace']/node()[substring(@name, 1, 8)='DemoType']",
            namespaces={"xs": "http://www.w3.org/2001/XMLSchema"}):
        defns.append(defn)
except:
    raise SystemExit(125)
else:
    types_text = etree.tostring(defns, pretty_print=True)
    print(types_text)
    assert "TypeType" not in types_text

Driving shell script:

#!/bin/sh

# Clean up old .pyc files
find -name \*.pyc -print0|xargs -0 rm -f
exec python array_demo.py

Expected output (like in spyne 2.9.3):

<matches xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:simpleType name="DemoType">
    <xs:restriction base="xs:string">
      <xs:pattern value="demo"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:complexType name="DemoTypeArray">
    <xs:sequence>
      <xs:element name="DemoType" type="tns:DemoType" minOccurs="0" maxOccurs="unbounded" nillable="true"/>
    </xs:sequence>
  </xs:complexType>
  <xs:element name="DemoTypeArray" type="tns:DemoTypeArray"/>
</matches>

Actual output (in spyne 2.12.14):

<matches xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:simpleType name="DemoType">
    <xs:restriction base="xs:string">
      <xs:pattern value="demo"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="DemoTypeArray_DemoTypeType">
    <xs:restriction base="xs:string">
      <xs:pattern value="demo"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:complexType name="DemoTypeArray">
    <xs:sequence>
      <xs:element name="DemoType" type="tns:DemoTypeArray_DemoTypeType" minOccurs="0" maxOccurs="unbounded" nillable="true"/>
    </xs:sequence>
  </xs:complexType>
  <xs:element name="DemoTypeArray" type="tns:DemoTypeArray"/>
</matches>

Traceback (most recent call last):
  File "array_demo.py", line 52, in <module>
    assert "TypeType" not in types_text
AssertionError

[tlandschoff-scale]

This seems to boil down to this behaviour wrt. customize:

$ python
Python 2.7.12 (default, Nov  4 2017, 00:30:53) 
>>> import decimal
>>> from spyne import Uuid
>>> from spyne.model.primitive import Unicode
>>> 
>>> UnboundedUnicode = Unicode.customize(max_occurs=decimal.Decimal("inf"))
>>> UnboundedUnicode.is_default(UnboundedUnicode)
True
>>> UnboundedUUID = Uuid.customize(max_occurs=decimal.Decimal("inf"))
>>> UnboundedUUID.is_default(UnboundedUUID)
False
>>> DemoType = Unicode(type_name="DemoType", pattern="demo")
>>> UnboundedDemoType = DemoType(max_occurs=decimal.Decimal("inf"))
>>> UnboundedDemoType.is_default(UnboundedDemoType)
False

When defining a type by assigning

DemoArray = Array(DemoType)

the item type is customized under the hood by this code:

        # hack to default to unbounded arrays when the user didn't specify
        # max_occurs.
        if serializer.Attributes.max_occurs == 1:
            serializer = serializer.customize(max_occurs=decimal.Decimal('inf'))

Apart from this making no sense to me because it does not compose (making an array of some type with max_occurs=3 will drop that max_occurs, but I think this is broken in XML Schema already), this leads to the creation of a new type name in customize due to this code:

        retval = type(cls_name, cls_bases, cls_dict)

        if not retval.is_default(retval):
            retval.__extends__ = cls
            ...

I'd imagine this basically hinders reuse of any custom type in arrays.

IMHO the method is_default is badly named. It basically checks if a SimpleModel is equivalent to its base. Moreover, the base is not given so it checks against the wrong base when checking customized types of customized types.

[plq]

First, thanks a lot for digging. It seems that the WSDL output is not strictly wrong but suboptimal and could be fixed (hence the enhancement flag). Would you agree?

the method is_default is badly named. It basically checks if a SimpleModel is equivalent to its base. Moreover, the base is not given so it checks against the wrong base when checking customized types of customized types.

That's by design. is_default is supposed to compare a given type to its root type, not its immediate base type. Uuid being a fancy string (as far as Spyne is concerned) makes its default configuration a non-default Unicode.

the item type is customized under the hood by this code:

(...)

Apart from this making no sense to me because it does not compose (making an array of some type with max_occurs=3 will drop that max_occurs, but I think this is broken in XML Schema already), this leads to the creation of a new type name in customize due to this code:

Again, this is by design. Array(Foo) means Array(Foo.customize(max_occurs='unbounded)).

[plq]

Instead of looking at the model subsystem, I think we should look at the xml schema subsystem.

In the xml schema world, there are two types of customizations: stuff like minOccurs, maxOccurs, etc that can be made at the element level and stuff like maxLength that can be made at the type level.

The strategy here should be to distinguish between these two types of customizations and replace types that only have element-level customizations (compared to its immediate base, not its root) while generating the xml schema.

Here's the entry point, which should end up calling spyne.interface.xml_schema.model.complex_add():

spyne/spyne/interface/xml_schema/_base.py

Line 117 in 049b789

def add(self, cls, tags):

[tlandschoff-scale]

AFAICT this is_default basically exists solely for the purpose of filtering duplicate types.
Therefore I'd think this should work or be dropped altogether , leaving it to the XML generation layer to filter duplicate definitions.

I am not sure though if that information has to be channeled back from the XML layer - I would expect that to be required.

My approach was to improve is_default to actually detect if a customized type is essentially the same as the original type. This fixed my issue but makes a number of other tests fail.

[plq]

Instead of messing with is_default, you should implement the same test in the xml schema generator so that it ignores the "duplicate" type and uses the original one.

[gaojingwei]

I have encountered the same problem. How do you solve it in the end?

[tlandschoff-scale]

We forked at spyne 2.9.3 and ported it to Python 3 ourselves (at least in so far that our unit and integration tests are running).

As we are no longer required to use SOAP as the communication protocol we are slowly moving away from spyne. It has a lot of magic that has little benefit to us as we will have a single RESTful interface.

But we may return to spyne in case some client dictates that we have to support another protocol again.

Full disclosure: We started with using soaplib as it appeared to be the best soap library for Python on the server side. We migrated to rpclib and later spyne still using only the soap part.

[gaojingwei]

I slove the problem by http://spyne.io/docs/2.10/manual/03_types.html#arrays
if use:
permissions = Permission.customize(max_occurs='unbounded')
or
Array( permissions)
it will generate the extr type PermissionArray, but if use
permissions = Permission.customize(max_occurs=float('inf'))
it will run well, and not generate PermissionArray