One of the things I like about the way IS does validation is that the parser just does XML syntax validation (the minimal amount of validation it can do to create a parse tree). All DTD/XSD or pipeline validation is done in layers above the parser, allowing the application level to decide how to handle schema errors.
If your XML is so bad it fails documentToRecord then no parser would be able to create a parse tree from it, but since the default mode of the Integration Server XML parser is streaming (true since 3.0), even the parse errors are often delayed until the application server has control.
How you handle it is up to what you want your application behavior to be. If you want to return a specific different error for XML syntax errors and schema validation errors you can. You could try to recover from certain schema validation errors. Or call pub.schema:validate on the xmlNode and make sure everything is OK.
#Integration-Server-and-ESB#Flow-and-Java-services#webMethods